2023-10-14T11:46:04.663 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-14T11:46:04.664 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_rishabh@teuthology --description fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} --archive /home/teuthworker/archive/rishabh-2023-10-13_18:04:14-fs-rishabh-2023oct13-testing-default-smithi/7425292 --name rishabh-2023-10-13_18:04:14-fs-rishabh-2023oct13-testing-default-smithi -- /home/teuthworker/archive/rishabh-2023-10-13_18:04:14-fs-rishabh-2023oct13-testing-default-smithi/7425292/orig.config.yaml 2023-10-14T11:46:04.708 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-14T11:46:04.791 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2023-10-13_18:04:14-fs-rishabh-2023oct13-testing-default-smithi/7425292 branch: rishabh-2023oct13 description: fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} email: null first_in_suite: false job_id: '7425292' kernel: client: sha1: distro flavor: default kdb: true sha1: 47672119149b103ec7cec2cc65c063ccbb001ac7 ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: rishabh-2023-10-13_18:04:14-fs-rishabh-2023oct13-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: rishabh-2023oct13 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - Replacing daemon mds - \(MDS_INSUFFICIENT_STANDBY\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(MDS_DAMAGE\) - \(FS_DEGRADED\) sha1: 4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: disabled: true install: ceph: flavor: default sha1: 4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0 kclient: syntax: v2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: rishabh-2023oct13 sha1: 4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0 owner: scheduled_rishabh@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 seed: 6006 sha1: 4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0 sleep_before_teardown: 0 subset: 402/512 suite: fs suite_branch: rishabh-2023oct13 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi078.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= smithi195.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - exec: mon.a: - ceph config set mgr mgr/crash/warn_recent_interval 0 - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_failover teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 8cdab074dcca9a68965bc5a50e9c30b691949723 timestamp: 2023-10-13_18:04:14 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-14T11:46:04.792 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/qa; will attempt to use it 2023-10-14T11:46:04.793 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/qa/tasks 2023-10-14T11:46:04.793 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-14T11:46:04.795 INFO:teuthology.task.internal:Checking packages... 2023-10-14T11:46:04.818 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0' 2023-10-14T11:46:04.819 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-14T11:46:04.819 INFO:teuthology.packaging:ref: None 2023-10-14T11:46:04.820 INFO:teuthology.packaging:tag: None 2023-10-14T11:46:04.820 INFO:teuthology.packaging:branch: rishabh-2023oct13 2023-10-14T11:46:04.820 INFO:teuthology.packaging:sha1: 4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0 2023-10-14T11:46:04.821 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=rishabh-2023oct13 2023-10-14T11:46:04.967 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6730.g4ba8a190 2023-10-14T11:46:04.968 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-14T11:46:04.976 INFO:teuthology.task.internal:no buildpackages task found 2023-10-14T11:46:04.977 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-14T11:46:04.994 INFO:teuthology.task.internal:Saving configuration 2023-10-14T11:46:05.009 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-14T11:46:05.018 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-14T11:46:05.042 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi078.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-10-13_18:04:14-fs-rishabh-2023oct13-testing-default-smithi/7425292', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-14 11:40:59.830502', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-10-14T11:46:05.064 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi195.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-10-13_18:04:14-fs-rishabh-2023oct13-testing-default-smithi/7425292', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-14 11:40:59.832506', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-10-14T11:46:05.064 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-14T11:46:05.075 INFO:teuthology.task.internal:roles: ubuntu@smithi078.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2023-10-14T11:46:05.075 INFO:teuthology.task.internal:roles: ubuntu@smithi195.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2023-10-14T11:46:05.076 INFO:teuthology.run_tasks:Running task console_log... 2023-10-14T11:46:05.156 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa7654aaaf0>, signals=[15]) 2023-10-14T11:46:05.157 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-14T11:46:05.169 INFO:teuthology.task.internal:Opening connections... 2023-10-14T11:46:05.170 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi078.front.sepia.ceph.com 2023-10-14T11:46:05.171 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-14T11:46:05.239 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi195.front.sepia.ceph.com 2023-10-14T11:46:05.240 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-14T11:46:05.306 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-14T11:46:05.313 DEBUG:teuthology.orchestra.run.smithi078:> uname -m 2023-10-14T11:46:05.335 INFO:teuthology.orchestra.run.smithi078.stdout:x86_64 2023-10-14T11:46:05.336 DEBUG:teuthology.orchestra.run.smithi078:> cat /etc/os-release 2023-10-14T11:46:05.396 INFO:teuthology.orchestra.run.smithi078.stdout:NAME="Red Hat Enterprise Linux" 2023-10-14T11:46:05.396 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION="8.6 (Ootpa)" 2023-10-14T11:46:05.396 INFO:teuthology.orchestra.run.smithi078.stdout:ID="rhel" 2023-10-14T11:46:05.396 INFO:teuthology.orchestra.run.smithi078.stdout:ID_LIKE="fedora" 2023-10-14T11:46:05.397 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_ID="8.6" 2023-10-14T11:46:05.397 INFO:teuthology.orchestra.run.smithi078.stdout:PLATFORM_ID="platform:el8" 2023-10-14T11:46:05.397 INFO:teuthology.orchestra.run.smithi078.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-10-14T11:46:05.397 INFO:teuthology.orchestra.run.smithi078.stdout:ANSI_COLOR="0;31" 2023-10-14T11:46:05.398 INFO:teuthology.orchestra.run.smithi078.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-10-14T11:46:05.398 INFO:teuthology.orchestra.run.smithi078.stdout:HOME_URL="https://www.redhat.com/" 2023-10-14T11:46:05.398 INFO:teuthology.orchestra.run.smithi078.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-14T11:46:05.398 INFO:teuthology.orchestra.run.smithi078.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-14T11:46:05.398 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:46:05.399 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-14T11:46:05.399 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-10-14T11:46:05.399 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-14T11:46:05.399 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-10-14T11:46:05.400 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server 2023-10-14T11:46:05.425 DEBUG:teuthology.orchestra.run.smithi195:> uname -m 2023-10-14T11:46:05.445 INFO:teuthology.orchestra.run.smithi195.stdout:x86_64 2023-10-14T11:46:05.446 DEBUG:teuthology.orchestra.run.smithi195:> cat /etc/os-release 2023-10-14T11:46:05.506 INFO:teuthology.orchestra.run.smithi195.stdout:NAME="Red Hat Enterprise Linux" 2023-10-14T11:46:05.506 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION="8.6 (Ootpa)" 2023-10-14T11:46:05.507 INFO:teuthology.orchestra.run.smithi195.stdout:ID="rhel" 2023-10-14T11:46:05.507 INFO:teuthology.orchestra.run.smithi195.stdout:ID_LIKE="fedora" 2023-10-14T11:46:05.507 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION_ID="8.6" 2023-10-14T11:46:05.507 INFO:teuthology.orchestra.run.smithi195.stdout:PLATFORM_ID="platform:el8" 2023-10-14T11:46:05.507 INFO:teuthology.orchestra.run.smithi195.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-10-14T11:46:05.508 INFO:teuthology.orchestra.run.smithi195.stdout:ANSI_COLOR="0;31" 2023-10-14T11:46:05.508 INFO:teuthology.orchestra.run.smithi195.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-10-14T11:46:05.508 INFO:teuthology.orchestra.run.smithi195.stdout:HOME_URL="https://www.redhat.com/" 2023-10-14T11:46:05.508 INFO:teuthology.orchestra.run.smithi195.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-14T11:46:05.508 INFO:teuthology.orchestra.run.smithi195.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-14T11:46:05.509 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-14T11:46:05.509 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-14T11:46:05.509 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-10-14T11:46:05.509 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-14T11:46:05.509 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-10-14T11:46:05.510 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server 2023-10-14T11:46:05.534 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-14T11:46:05.545 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-14T11:46:05.552 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-14T11:46:05.553 DEBUG:teuthology.orchestra.run.smithi078:> test '!' -e /home/ubuntu/cephtest 2023-10-14T11:46:05.556 DEBUG:teuthology.orchestra.run.smithi195:> test '!' -e /home/ubuntu/cephtest 2023-10-14T11:46:05.575 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-14T11:46:05.582 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-14T11:46:05.582 DEBUG:teuthology.orchestra.run.smithi078:> test -z $(ls -A /var/lib/ceph) 2023-10-14T11:46:05.615 DEBUG:teuthology.orchestra.run.smithi195:> test -z $(ls -A /var/lib/ceph) 2023-10-14T11:46:05.645 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-14T11:46:05.703 INFO:teuthology.run_tasks:Running task kernel... 2023-10-14T11:46:05.719 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '47672119149b103ec7cec2cc65c063ccbb001ac7'} 2023-10-14T11:46:05.719 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}} 2023-10-14T11:46:05.720 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2023-10-14T11:46:05.720 DEBUG:teuthology.orchestra.run.smithi078:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-14T11:46:05.721 DEBUG:teuthology.orchestra.run.smithi195:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-14T11:46:05.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T11:46:05.741 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2023-10-14T11:46:05.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T11:46:05.742 DEBUG:teuthology.orchestra.run.smithi195:> uname -r 2023-10-14T11:46:05.801 INFO:teuthology.orchestra.run.smithi195.stdout:4.18.0-372.9.1.el8.x86_64 2023-10-14T11:46:05.802 INFO:teuthology.orchestra.run.smithi078.stdout:4.18.0-372.9.1.el8.x86_64 2023-10-14T11:46:05.802 INFO:teuthology.task.kernel:Running kernel on smithi195: 4.18.0-372.9.1.el8.x86_64 2023-10-14T11:46:05.802 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum install -y kernel 2023-10-14T11:46:05.803 INFO:teuthology.task.kernel:Running kernel on smithi078: 4.18.0-372.9.1.el8.x86_64 2023-10-14T11:46:05.803 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum install -y kernel 2023-10-14T11:46:06.970 INFO:teuthology.orchestra.run.smithi195.stdout:Updating Subscription Management repositories. 2023-10-14T11:46:06.970 INFO:teuthology.orchestra.run.smithi195.stdout:Unable to read consumer identity 2023-10-14T11:46:07.120 INFO:teuthology.orchestra.run.smithi078.stdout:Updating Subscription Management repositories. 2023-10-14T11:46:07.121 INFO:teuthology.orchestra.run.smithi078.stdout:Unable to read consumer identity 2023-10-14T11:46:08.389 INFO:teuthology.orchestra.run.smithi195.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2023-10-14T11:46:08.584 INFO:teuthology.orchestra.run.smithi078.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2023-10-14T11:46:09.056 INFO:teuthology.orchestra.run.smithi195.stdout:Extra Packages for Enterprise Linux 25 MB/s | 16 MB 00:00 2023-10-14T11:46:10.176 INFO:teuthology.orchestra.run.smithi078.stdout:Extra Packages for Enterprise Linux 10 MB/s | 16 MB 00:01 2023-10-14T11:46:12.506 INFO:teuthology.orchestra.run.smithi195.stdout:lab-extras 407 kB/s | 24 kB 00:00 2023-10-14T11:46:13.184 INFO:teuthology.orchestra.run.smithi195.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-10-14T11:46:13.251 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2023-10-14T11:46:13.252 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2023-10-14T11:46:13.253 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2023-10-14T11:46:13.380 DEBUG:teuthology.orchestra.run.smithi195:> echo no | sudo yum reinstall kernel || true 2023-10-14T11:46:13.709 INFO:teuthology.orchestra.run.smithi195.stdout:Updating Subscription Management repositories. 2023-10-14T11:46:13.709 INFO:teuthology.orchestra.run.smithi195.stdout:Unable to read consumer identity 2023-10-14T11:46:13.735 INFO:teuthology.orchestra.run.smithi078.stdout:lab-extras 281 kB/s | 24 kB 00:00 2023-10-14T11:46:13.974 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:00:01 ago on Sat 14 Oct 2023 11:46:12 AM UTC. 2023-10-14T11:46:14.017 INFO:teuthology.orchestra.run.smithi195.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-10-14T11:46:14.048 INFO:teuthology.orchestra.run.smithi195.stderr:Error: No packages marked for reinstall. 2023-10-14T11:46:14.094 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum reinstall -y kernel || true 2023-10-14T11:46:14.423 INFO:teuthology.orchestra.run.smithi195.stdout:Updating Subscription Management repositories. 2023-10-14T11:46:14.423 INFO:teuthology.orchestra.run.smithi195.stdout:Unable to read consumer identity 2023-10-14T11:46:14.446 INFO:teuthology.orchestra.run.smithi078.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-10-14T11:46:14.491 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-10-14T11:46:14.492 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2023-10-14T11:46:14.492 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-10-14T11:46:14.589 DEBUG:teuthology.orchestra.run.smithi078:> echo no | sudo yum reinstall kernel || true 2023-10-14T11:46:14.683 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:00:02 ago on Sat 14 Oct 2023 11:46:12 AM UTC. 2023-10-14T11:46:14.725 INFO:teuthology.orchestra.run.smithi195.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-10-14T11:46:14.752 INFO:teuthology.orchestra.run.smithi195.stderr:Error: No packages marked for reinstall. 2023-10-14T11:46:14.799 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q kernel | sort -rV | head -n 1 2023-10-14T11:46:14.879 INFO:teuthology.orchestra.run.smithi195.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-10-14T11:46:14.879 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-10-14T11:46:14.880 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-10-14T11:46:14.880 DEBUG:teuthology.parallel:result is None 2023-10-14T11:46:14.919 INFO:teuthology.orchestra.run.smithi078.stdout:Updating Subscription Management repositories. 2023-10-14T11:46:14.919 INFO:teuthology.orchestra.run.smithi078.stdout:Unable to read consumer identity 2023-10-14T11:46:15.182 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:00:02 ago on Sat 14 Oct 2023 11:46:13 AM UTC. 2023-10-14T11:46:15.223 INFO:teuthology.orchestra.run.smithi078.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-10-14T11:46:15.251 INFO:teuthology.orchestra.run.smithi078.stderr:Error: No packages marked for reinstall. 2023-10-14T11:46:15.297 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum reinstall -y kernel || true 2023-10-14T11:46:15.625 INFO:teuthology.orchestra.run.smithi078.stdout:Updating Subscription Management repositories. 2023-10-14T11:46:15.626 INFO:teuthology.orchestra.run.smithi078.stdout:Unable to read consumer identity 2023-10-14T11:46:15.886 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:00:02 ago on Sat 14 Oct 2023 11:46:13 AM UTC. 2023-10-14T11:46:15.926 INFO:teuthology.orchestra.run.smithi078.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-10-14T11:46:15.949 INFO:teuthology.orchestra.run.smithi078.stderr:Error: No packages marked for reinstall. 2023-10-14T11:46:15.995 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q kernel | sort -rV | head -n 1 2023-10-14T11:46:16.075 INFO:teuthology.orchestra.run.smithi078.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-10-14T11:46:16.075 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-10-14T11:46:16.076 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-10-14T11:46:16.076 DEBUG:teuthology.parallel:result is None 2023-10-14T11:46:16.076 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-14T11:46:16.085 INFO:teuthology.task.internal:Creating test directory... 2023-10-14T11:46:16.085 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-14T11:46:16.088 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-14T11:46:16.109 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-14T11:46:16.117 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-14T11:46:16.130 INFO:teuthology.task.internal:Creating archive directory... 2023-10-14T11:46:16.131 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-14T11:46:16.145 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-14T11:46:16.204 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-14T11:46:16.212 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-14T11:46:16.213 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-14T11:46:16.215 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-14T11:46:16.257 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-14T11:46:16.261 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-14T11:46:16.272 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-14T11:46:16.274 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-14T11:46:16.275 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-14T11:46:16.284 INFO:teuthology.task.internal:Configuring sudo... 2023-10-14T11:46:16.284 DEBUG:teuthology.orchestra.run.smithi078:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-14T11:46:16.316 DEBUG:teuthology.orchestra.run.smithi195:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-14T11:46:16.345 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-14T11:46:16.357 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-14T11:46:16.357 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-14T11:46:16.382 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-14T11:46:16.403 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-14T11:46:16.480 DEBUG:teuthology.orchestra.run.smithi078:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-14T11:46:16.541 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-14T11:46:16.585 DEBUG:teuthology.orchestra.run.smithi078:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-14T11:46:16.626 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-14T11:46:16.626 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-14T11:46:16.692 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-14T11:46:16.742 DEBUG:teuthology.orchestra.run.smithi195:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-14T11:46:16.790 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-14T11:46:16.873 DEBUG:teuthology.orchestra.run.smithi195:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-14T11:46:16.920 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-14T11:46:16.920 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-14T11:46:16.990 DEBUG:teuthology.orchestra.run.smithi078:> sudo service rsyslog restart 2023-10-14T11:46:16.992 DEBUG:teuthology.orchestra.run.smithi195:> sudo service rsyslog restart 2023-10-14T11:46:17.080 INFO:teuthology.orchestra.run.smithi195.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-14T11:46:17.089 INFO:teuthology.orchestra.run.smithi078.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-14T11:46:17.351 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-14T11:46:17.359 INFO:teuthology.task.internal:Starting timer... 2023-10-14T11:46:17.359 INFO:teuthology.run_tasks:Running task pcp... 2023-10-14T11:46:17.380 INFO:teuthology.run_tasks:Running task selinux... 2023-10-14T11:46:17.390 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-10-14T11:46:17.391 DEBUG:teuthology.orchestra.run.smithi078:> sudo service auditd rotate 2023-10-14T11:46:17.462 INFO:teuthology.orchestra.run.smithi078.stdout:Rotating logs: 2023-10-14T11:46:17.464 DEBUG:teuthology.orchestra.run.smithi195:> sudo service auditd rotate 2023-10-14T11:46:17.524 INFO:teuthology.orchestra.run.smithi195.stdout:Rotating logs: 2023-10-14T11:46:17.525 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-14T11:46:17.526 DEBUG:teuthology.orchestra.run.smithi078:> /usr/sbin/getenforce 2023-10-14T11:46:17.548 INFO:teuthology.orchestra.run.smithi078.stdout:Permissive 2023-10-14T11:46:17.548 DEBUG:teuthology.orchestra.run.smithi195:> /usr/sbin/getenforce 2023-10-14T11:46:17.572 INFO:teuthology.orchestra.run.smithi195.stdout:Permissive 2023-10-14T11:46:17.572 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi078.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi195.front.sepia.ceph.com': 'permissive'} 2023-10-14T11:46:17.573 DEBUG:teuthology.orchestra.run.smithi078:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-14T11:46:17.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T11:46:17.614 DEBUG:teuthology.orchestra.run.smithi195:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-14T11:46:17.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T11:46:17.644 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-14T11:46:17.645 DEBUG:teuthology.orchestra.run.smithi078:> sudo /usr/sbin/setenforce permissive 2023-10-14T11:46:17.699 DEBUG:teuthology.orchestra.run.smithi195:> sudo /usr/sbin/setenforce permissive 2023-10-14T11:46:17.727 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-14T11:46:17.737 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-14T11:46:17.742 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-14T11:46:17.865 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-14T11:46:17.885 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-14T11:46:17.887 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi078.front.sepia.ceph.com,smithi195.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-14T11:55:04.310 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi078.front.sepia.ceph.com'), Remote(name='ubuntu@smithi195.front.sepia.ceph.com')] 2023-10-14T11:55:04.312 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2023-10-14T11:55:04.313 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-14T11:55:04.382 DEBUG:teuthology.orchestra.run.smithi078:> true 2023-10-14T11:55:04.451 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2023-10-14T11:55:04.452 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2023-10-14T11:55:04.452 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-14T11:55:04.521 DEBUG:teuthology.orchestra.run.smithi195:> true 2023-10-14T11:55:04.590 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi195.front.sepia.ceph.com' 2023-10-14T11:55:04.591 INFO:teuthology.run_tasks:Running task clock... 2023-10-14T11:55:04.602 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-14T11:55:04.602 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-14T11:55:04.603 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-14T11:55:04.606 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-14T11:55:04.606 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-14T11:55:04.634 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-14T11:55:04.649 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-14T11:55:04.664 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-14T11:55:04.680 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-14T11:55:04.711 INFO:teuthology.orchestra.run.smithi078.stderr:sudo: ntpd: command not found 2023-10-14T11:55:04.722 INFO:teuthology.orchestra.run.smithi078.stdout:506 Cannot talk to daemon 2023-10-14T11:55:04.737 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-14T11:55:04.746 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: ntpd: command not found 2023-10-14T11:55:04.750 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-14T11:55:04.758 INFO:teuthology.orchestra.run.smithi195.stdout:506 Cannot talk to daemon 2023-10-14T11:55:04.772 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-14T11:55:04.786 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-14T11:55:04.819 INFO:teuthology.orchestra.run.smithi078.stderr:bash: ntpq: command not found 2023-10-14T11:55:04.823 INFO:teuthology.orchestra.run.smithi078.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-14T11:55:04.823 INFO:teuthology.orchestra.run.smithi078.stdout:=============================================================================== 2023-10-14T11:55:04.823 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-14T11:55:04.823 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-14T11:55:04.824 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-14T11:55:04.824 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-14T11:55:04.864 INFO:teuthology.orchestra.run.smithi195.stderr:bash: ntpq: command not found 2023-10-14T11:55:04.868 INFO:teuthology.orchestra.run.smithi195.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-14T11:55:04.868 INFO:teuthology.orchestra.run.smithi195.stdout:=============================================================================== 2023-10-14T11:55:04.869 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-14T11:55:04.869 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-14T11:55:04.869 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-14T11:55:04.869 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-14T11:55:04.870 INFO:teuthology.run_tasks:Running task install... 2023-10-14T11:55:04.878 DEBUG:teuthology.task.install:project ceph 2023-10-14T11:55:04.879 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0'}} 2023-10-14T11:55:04.879 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0'} 2023-10-14T11:55:04.879 INFO:teuthology.task.install:Using flavor: default 2023-10-14T11:55:04.884 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-14T11:55:04.885 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-10-14T11:55:04.885 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0', 'tag': None, 'wait_for_package': False} 2023-10-14T11:55:04.886 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0 2023-10-14T11:55:04.888 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0', 'tag': None, 'wait_for_package': False} 2023-10-14T11:55:04.888 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0 2023-10-14T11:55:05.111 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/centos/8/flavors/default/ 2023-10-14T11:55:05.111 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6730.g4ba8a190 2023-10-14T11:55:05.117 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/centos/8/flavors/default/ 2023-10-14T11:55:05.117 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6730.g4ba8a190 2023-10-14T11:55:05.219 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-14T11:55:05.219 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-14T11:55:05.220 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-14T11:55:05.233 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-14T11:55:05.233 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-14T11:55:05.234 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-14T11:55:05.248 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-10-14T11:55:05.248 DEBUG:teuthology.orchestra.run.smithi195:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-14T11:55:05.263 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-10-14T11:55:05.264 DEBUG:teuthology.orchestra.run.smithi078:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-14T11:55:05.315 DEBUG:teuthology.orchestra.run.smithi195:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-14T11:55:05.332 DEBUG:teuthology.orchestra.run.smithi078:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-14T11:55:05.392 DEBUG:teuthology.orchestra.run.smithi195:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-14T11:55:05.410 DEBUG:teuthology.orchestra.run.smithi078:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-14T11:55:05.457 INFO:teuthology.orchestra.run.smithi195.stdout:check_obsoletes = 1 2023-10-14T11:55:05.458 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum clean all 2023-10-14T11:55:05.476 INFO:teuthology.orchestra.run.smithi078.stdout:check_obsoletes = 1 2023-10-14T11:55:05.477 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum clean all 2023-10-14T11:55:05.826 INFO:teuthology.orchestra.run.smithi195.stdout:Updating Subscription Management repositories. 2023-10-14T11:55:05.847 INFO:teuthology.orchestra.run.smithi078.stdout:Updating Subscription Management repositories. 2023-10-14T11:55:06.103 INFO:teuthology.orchestra.run.smithi195.stdout:44 files removed 2023-10-14T11:55:06.150 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-10-14T11:55:06.153 INFO:teuthology.orchestra.run.smithi078.stdout:44 files removed 2023-10-14T11:55:06.199 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-10-14T11:55:06.480 INFO:teuthology.orchestra.run.smithi195.stdout:Updating Subscription Management repositories. 2023-10-14T11:55:06.528 INFO:teuthology.orchestra.run.smithi078.stdout:Updating Subscription Management repositories. 2023-10-14T11:55:07.244 INFO:teuthology.orchestra.run.smithi078.stdout:ceph packages for x86_64 271 kB/s | 81 kB 00:00 2023-10-14T11:55:07.245 INFO:teuthology.orchestra.run.smithi195.stdout:ceph packages for x86_64 228 kB/s | 81 kB 00:00 2023-10-14T11:55:07.580 INFO:teuthology.orchestra.run.smithi195.stdout:ceph noarch packages 31 kB/s | 9.4 kB 00:00 2023-10-14T11:55:07.604 INFO:teuthology.orchestra.run.smithi078.stdout:ceph noarch packages 29 kB/s | 9.4 kB 00:00 2023-10-14T11:55:07.863 INFO:teuthology.orchestra.run.smithi078.stdout:ceph source packages 6.3 kB/s | 1.5 kB 00:00 2023-10-14T11:55:07.930 INFO:teuthology.orchestra.run.smithi195.stdout:ceph source packages 4.5 kB/s | 1.5 kB 00:00 2023-10-14T11:55:08.047 INFO:teuthology.orchestra.run.smithi078.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2023-10-14T11:55:08.081 INFO:teuthology.orchestra.run.smithi195.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2023-10-14T11:55:08.856 INFO:teuthology.orchestra.run.smithi195.stdout:Extra Packages for Enterprise Linux 21 MB/s | 16 MB 00:00 2023-10-14T11:55:09.385 INFO:teuthology.orchestra.run.smithi078.stdout:Extra Packages for Enterprise Linux 12 MB/s | 16 MB 00:01 2023-10-14T11:55:12.423 INFO:teuthology.orchestra.run.smithi195.stdout:lab-extras 518 kB/s | 24 kB 00:00 2023-10-14T11:55:12.898 INFO:teuthology.orchestra.run.smithi078.stdout:lab-extras 399 kB/s | 24 kB 00:00 2023-10-14T11:55:13.326 INFO:teuthology.orchestra.run.smithi195.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 57 MB/s | 47 MB 00:00 2023-10-14T11:55:13.759 INFO:teuthology.orchestra.run.smithi078.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 60 MB/s | 47 MB 00:00 2023-10-14T11:55:21.126 INFO:teuthology.orchestra.run.smithi195.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 66 MB/s | 53 MB 00:00 2023-10-14T11:55:21.501 INFO:teuthology.orchestra.run.smithi078.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 67 MB/s | 53 MB 00:00 2023-10-14T11:55:32.183 INFO:teuthology.orchestra.run.smithi195.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-14T11:55:32.184 INFO:teuthology.orchestra.run.smithi195.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-14T11:55:32.198 INFO:teuthology.orchestra.run.smithi195.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-10-14T11:55:32.199 INFO:teuthology.orchestra.run.smithi195.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-10-14T11:55:32.200 INFO:teuthology.orchestra.run.smithi195.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-10-14T11:55:32.201 INFO:teuthology.orchestra.run.smithi195.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-10-14T11:55:32.203 INFO:teuthology.orchestra.run.smithi195.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-10-14T11:55:32.207 INFO:teuthology.orchestra.run.smithi195.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-10-14T11:55:32.208 INFO:teuthology.orchestra.run.smithi195.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-10-14T11:55:32.338 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2023-10-14T11:55:32.350 INFO:teuthology.orchestra.run.smithi195.stdout:=========================================================================================================================== 2023-10-14T11:55:32.350 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2023-10-14T11:55:32.350 INFO:teuthology.orchestra.run.smithi195.stdout:=========================================================================================================================== 2023-10-14T11:55:32.351 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2023-10-14T11:55:32.351 INFO:teuthology.orchestra.run.smithi195.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-10-14T11:55:32.351 INFO:teuthology.orchestra.run.smithi195.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-10-14T11:55:32.351 INFO:teuthology.orchestra.run.smithi195.stdout: ceph x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 6.4 k 2023-10-14T11:55:32.352 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-base x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 5.4 M 2023-10-14T11:55:32.352 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-fuse x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 918 k 2023-10-14T11:55:32.352 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 167 k 2023-10-14T11:55:32.352 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 1.6 M 2023-10-14T11:55:32.352 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6730.g4ba8a190.el8 ceph-noarch 132 k 2023-10-14T11:55:32.353 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6730.g4ba8a190.el8 ceph-noarch 1.8 M 2023-10-14T11:55:32.353 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6730.g4ba8a190.el8 ceph-noarch 7.4 M 2023-10-14T11:55:32.353 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-rook noarch 2:18.0.0-6730.g4ba8a190.el8 ceph-noarch 47 k 2023-10-14T11:55:32.353 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-radosgw x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 14 M 2023-10-14T11:55:32.353 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-test x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 49 M 2023-10-14T11:55:32.354 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-volume noarch 2:18.0.0-6730.g4ba8a190.el8 ceph-noarch 261 k 2023-10-14T11:55:32.354 INFO:teuthology.orchestra.run.smithi195.stdout: cephadm noarch 2:18.0.0-6730.g4ba8a190.el8 ceph-noarch 192 k 2023-10-14T11:55:32.354 INFO:teuthology.orchestra.run.smithi195.stdout: cephfs-mirror x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 261 k 2023-10-14T11:55:32.354 INFO:teuthology.orchestra.run.smithi195.stdout: cephfs-top noarch 2:18.0.0-6730.g4ba8a190.el8 ceph-noarch 18 k 2023-10-14T11:55:32.354 INFO:teuthology.orchestra.run.smithi195.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-10-14T11:55:32.355 INFO:teuthology.orchestra.run.smithi195.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-10-14T11:55:32.355 INFO:teuthology.orchestra.run.smithi195.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-10-14T11:55:32.355 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs-devel x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 31 k 2023-10-14T11:55:32.355 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs2 x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 839 k 2023-10-14T11:55:32.355 INFO:teuthology.orchestra.run.smithi195.stdout: librados-devel x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 139 k 2023-10-14T11:55:32.356 INFO:teuthology.orchestra.run.smithi195.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-10-14T11:55:32.356 INFO:teuthology.orchestra.run.smithi195.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-10-14T11:55:32.356 INFO:teuthology.orchestra.run.smithi195.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-10-14T11:55:32.356 INFO:teuthology.orchestra.run.smithi195.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-10-14T11:55:32.356 INFO:teuthology.orchestra.run.smithi195.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-10-14T11:55:32.357 INFO:teuthology.orchestra.run.smithi195.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-10-14T11:55:32.357 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cephfs x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 197 k 2023-10-14T11:55:32.357 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rados x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 371 k 2023-10-14T11:55:32.357 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rbd x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 365 k 2023-10-14T11:55:32.357 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rgw x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 107 k 2023-10-14T11:55:32.358 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-fuse x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 94 k 2023-10-14T11:55:32.358 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-mirror x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 3.6 M 2023-10-14T11:55:32.358 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-nbd x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 179 k 2023-10-14T11:55:32.358 INFO:teuthology.orchestra.run.smithi195.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-10-14T11:55:32.358 INFO:teuthology.orchestra.run.smithi195.stdout:Upgrading: 2023-10-14T11:55:32.359 INFO:teuthology.orchestra.run.smithi195.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-10-14T11:55:32.359 INFO:teuthology.orchestra.run.smithi195.stdout: librados2 x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 3.7 M 2023-10-14T11:55:32.359 INFO:teuthology.orchestra.run.smithi195.stdout: librbd1 x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 3.7 M 2023-10-14T11:55:32.359 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2023-10-14T11:55:32.359 INFO:teuthology.orchestra.run.smithi195.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-10-14T11:55:32.360 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-common x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 23 M 2023-10-14T11:55:32.360 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6730.g4ba8a190.el8 ceph-noarch 23 k 2023-10-14T11:55:32.360 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mds x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 2.3 M 2023-10-14T11:55:32.360 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6730.g4ba8a190.el8 ceph-noarch 242 k 2023-10-14T11:55:32.360 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mon x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 4.6 M 2023-10-14T11:55:32.361 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-osd x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 18 M 2023-10-14T11:55:32.361 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6730.g4ba8a190.el8 ceph-noarch 15 k 2023-10-14T11:55:32.361 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-selinux x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 24 k 2023-10-14T11:55:32.361 INFO:teuthology.orchestra.run.smithi195.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-10-14T11:55:32.362 INFO:teuthology.orchestra.run.smithi195.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-14T11:55:32.362 INFO:teuthology.orchestra.run.smithi195.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-10-14T11:55:32.362 INFO:teuthology.orchestra.run.smithi195.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-10-14T11:55:32.362 INFO:teuthology.orchestra.run.smithi195.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-10-14T11:55:32.362 INFO:teuthology.orchestra.run.smithi195.stdout: libcephsqlite x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 184 k 2023-10-14T11:55:32.363 INFO:teuthology.orchestra.run.smithi195.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-10-14T11:55:32.363 INFO:teuthology.orchestra.run.smithi195.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-10-14T11:55:32.363 INFO:teuthology.orchestra.run.smithi195.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-14T11:55:32.363 INFO:teuthology.orchestra.run.smithi195.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-10-14T11:55:32.363 INFO:teuthology.orchestra.run.smithi195.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-10-14T11:55:32.364 INFO:teuthology.orchestra.run.smithi195.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-10-14T11:55:32.364 INFO:teuthology.orchestra.run.smithi195.stdout: libradosstriper1 x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 518 k 2023-10-14T11:55:32.364 INFO:teuthology.orchestra.run.smithi195.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-10-14T11:55:32.364 INFO:teuthology.orchestra.run.smithi195.stdout: librgw2 x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 6.9 M 2023-10-14T11:55:32.364 INFO:teuthology.orchestra.run.smithi195.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-10-14T11:55:32.365 INFO:teuthology.orchestra.run.smithi195.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-10-14T11:55:32.365 INFO:teuthology.orchestra.run.smithi195.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-14T11:55:32.365 INFO:teuthology.orchestra.run.smithi195.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-10-14T11:55:32.365 INFO:teuthology.orchestra.run.smithi195.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-10-14T11:55:32.365 INFO:teuthology.orchestra.run.smithi195.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-10-14T11:55:32.366 INFO:teuthology.orchestra.run.smithi195.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-10-14T11:55:32.366 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-10-14T11:55:32.366 INFO:teuthology.orchestra.run.smithi195.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-10-14T11:55:32.366 INFO:teuthology.orchestra.run.smithi195.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-10-14T11:55:32.367 INFO:teuthology.orchestra.run.smithi195.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-10-14T11:55:32.367 INFO:teuthology.orchestra.run.smithi195.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-10-14T11:55:32.367 INFO:teuthology.orchestra.run.smithi195.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-10-14T11:55:32.367 INFO:teuthology.orchestra.run.smithi195.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-14T11:55:32.367 INFO:teuthology.orchestra.run.smithi195.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-14T11:55:32.367 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-14T11:55:32.368 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-argparse x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 45 k 2023-10-14T11:55:32.368 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-common x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 126 k 2023-10-14T11:55:32.368 INFO:teuthology.orchestra.run.smithi195.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-14T11:55:32.368 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-10-14T11:55:32.368 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-14T11:55:32.369 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-14T11:55:32.369 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-10-14T11:55:32.369 INFO:teuthology.orchestra.run.smithi195.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-14T11:55:32.369 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-14T11:55:32.369 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-14T11:55:32.370 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-10-14T11:55:32.370 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-10-14T11:55:32.370 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-10-14T11:55:32.370 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-14T11:55:32.370 INFO:teuthology.orchestra.run.smithi195.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-14T11:55:32.371 INFO:teuthology.orchestra.run.smithi195.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-14T11:55:32.371 INFO:teuthology.orchestra.run.smithi195.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-10-14T11:55:32.371 INFO:teuthology.orchestra.run.smithi195.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-10-14T11:55:32.371 INFO:teuthology.orchestra.run.smithi195.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-14T11:55:32.371 INFO:teuthology.orchestra.run.smithi195.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-10-14T11:55:32.372 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-10-14T11:55:32.372 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-10-14T11:55:32.372 INFO:teuthology.orchestra.run.smithi195.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-10-14T11:55:32.372 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-14T11:55:32.372 INFO:teuthology.orchestra.run.smithi195.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-14T11:55:32.373 INFO:teuthology.orchestra.run.smithi195.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-10-14T11:55:32.373 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-10-14T11:55:32.373 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-10-14T11:55:32.373 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-10-14T11:55:32.373 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-10-14T11:55:32.374 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-10-14T11:55:32.374 INFO:teuthology.orchestra.run.smithi195.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-14T11:55:32.374 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-14T11:55:32.374 INFO:teuthology.orchestra.run.smithi195.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-14T11:55:32.374 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-14T11:55:32.375 INFO:teuthology.orchestra.run.smithi195.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-10-14T11:55:32.375 INFO:teuthology.orchestra.run.smithi195.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-14T11:55:32.375 INFO:teuthology.orchestra.run.smithi195.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-14T11:55:32.375 INFO:teuthology.orchestra.run.smithi195.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-14T11:55:32.375 INFO:teuthology.orchestra.run.smithi195.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-14T11:55:32.376 INFO:teuthology.orchestra.run.smithi195.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-14T11:55:32.376 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-14T11:55:32.376 INFO:teuthology.orchestra.run.smithi195.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-14T11:55:32.376 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-14T11:55:32.376 INFO:teuthology.orchestra.run.smithi195.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-10-14T11:55:32.377 INFO:teuthology.orchestra.run.smithi195.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-14T11:55:32.377 INFO:teuthology.orchestra.run.smithi195.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-10-14T11:55:32.377 INFO:teuthology.orchestra.run.smithi195.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-10-14T11:55:32.377 INFO:teuthology.orchestra.run.smithi195.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-10-14T11:55:32.377 INFO:teuthology.orchestra.run.smithi195.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-10-14T11:55:32.378 INFO:teuthology.orchestra.run.smithi195.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-10-14T11:55:32.378 INFO:teuthology.orchestra.run.smithi195.stdout:Enabling module streams: 2023-10-14T11:55:32.378 INFO:teuthology.orchestra.run.smithi195.stdout: postgresql 10 2023-10-14T11:55:32.378 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-14T11:55:32.379 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2023-10-14T11:55:32.379 INFO:teuthology.orchestra.run.smithi195.stdout:=========================================================================================================================== 2023-10-14T11:55:32.379 INFO:teuthology.orchestra.run.smithi195.stdout:Install 128 Packages 2023-10-14T11:55:32.379 INFO:teuthology.orchestra.run.smithi195.stdout:Upgrade 3 Packages 2023-10-14T11:55:32.379 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-14T11:55:32.380 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 209 M 2023-10-14T11:55:32.380 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2023-10-14T11:55:32.643 INFO:teuthology.orchestra.run.smithi195.stdout:(1/131): ceph-18.0.0-6730.g4ba8a190.el8.x86_64. 52 kB/s | 6.4 kB 00:00 2023-10-14T11:55:32.658 INFO:teuthology.orchestra.run.smithi078.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-14T11:55:32.659 INFO:teuthology.orchestra.run.smithi078.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-14T11:55:32.674 INFO:teuthology.orchestra.run.smithi078.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-10-14T11:55:32.675 INFO:teuthology.orchestra.run.smithi078.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-10-14T11:55:32.676 INFO:teuthology.orchestra.run.smithi078.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-10-14T11:55:32.677 INFO:teuthology.orchestra.run.smithi078.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-10-14T11:55:32.679 INFO:teuthology.orchestra.run.smithi078.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-10-14T11:55:32.682 INFO:teuthology.orchestra.run.smithi078.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-10-14T11:55:32.684 INFO:teuthology.orchestra.run.smithi078.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-10-14T11:55:32.815 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-10-14T11:55:32.827 INFO:teuthology.orchestra.run.smithi078.stdout:=========================================================================================================================== 2023-10-14T11:55:32.827 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2023-10-14T11:55:32.827 INFO:teuthology.orchestra.run.smithi078.stdout:=========================================================================================================================== 2023-10-14T11:55:32.827 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2023-10-14T11:55:32.828 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-10-14T11:55:32.828 INFO:teuthology.orchestra.run.smithi078.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-10-14T11:55:32.828 INFO:teuthology.orchestra.run.smithi078.stdout: ceph x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 6.4 k 2023-10-14T11:55:32.828 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-base x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 5.4 M 2023-10-14T11:55:32.828 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-fuse x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 918 k 2023-10-14T11:55:32.829 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 167 k 2023-10-14T11:55:32.829 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 1.6 M 2023-10-14T11:55:32.829 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6730.g4ba8a190.el8 ceph-noarch 132 k 2023-10-14T11:55:32.829 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6730.g4ba8a190.el8 ceph-noarch 1.8 M 2023-10-14T11:55:32.830 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6730.g4ba8a190.el8 ceph-noarch 7.4 M 2023-10-14T11:55:32.830 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-rook noarch 2:18.0.0-6730.g4ba8a190.el8 ceph-noarch 47 k 2023-10-14T11:55:32.830 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-radosgw x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 14 M 2023-10-14T11:55:32.830 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-test x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 49 M 2023-10-14T11:55:32.830 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-volume noarch 2:18.0.0-6730.g4ba8a190.el8 ceph-noarch 261 k 2023-10-14T11:55:32.831 INFO:teuthology.orchestra.run.smithi078.stdout: cephadm noarch 2:18.0.0-6730.g4ba8a190.el8 ceph-noarch 192 k 2023-10-14T11:55:32.831 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-mirror x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 261 k 2023-10-14T11:55:32.831 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-top noarch 2:18.0.0-6730.g4ba8a190.el8 ceph-noarch 18 k 2023-10-14T11:55:32.831 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-10-14T11:55:32.831 INFO:teuthology.orchestra.run.smithi078.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-10-14T11:55:32.832 INFO:teuthology.orchestra.run.smithi078.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-10-14T11:55:32.832 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs-devel x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 31 k 2023-10-14T11:55:32.832 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs2 x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 839 k 2023-10-14T11:55:32.832 INFO:teuthology.orchestra.run.smithi078.stdout: librados-devel x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 139 k 2023-10-14T11:55:32.832 INFO:teuthology.orchestra.run.smithi078.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-10-14T11:55:32.833 INFO:teuthology.orchestra.run.smithi078.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-10-14T11:55:32.833 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-10-14T11:55:32.833 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-10-14T11:55:32.833 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-10-14T11:55:32.833 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-10-14T11:55:32.834 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cephfs x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 197 k 2023-10-14T11:55:32.834 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rados x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 371 k 2023-10-14T11:55:32.834 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rbd x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 365 k 2023-10-14T11:55:32.834 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rgw x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 107 k 2023-10-14T11:55:32.834 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-fuse x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 94 k 2023-10-14T11:55:32.835 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-mirror x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 3.6 M 2023-10-14T11:55:32.835 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-nbd x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 179 k 2023-10-14T11:55:32.835 INFO:teuthology.orchestra.run.smithi078.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-10-14T11:55:32.835 INFO:teuthology.orchestra.run.smithi078.stdout:Upgrading: 2023-10-14T11:55:32.835 INFO:teuthology.orchestra.run.smithi078.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-10-14T11:55:32.836 INFO:teuthology.orchestra.run.smithi078.stdout: librados2 x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 3.7 M 2023-10-14T11:55:32.836 INFO:teuthology.orchestra.run.smithi078.stdout: librbd1 x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 3.7 M 2023-10-14T11:55:32.836 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2023-10-14T11:55:32.836 INFO:teuthology.orchestra.run.smithi078.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-10-14T11:55:32.836 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-common x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 23 M 2023-10-14T11:55:32.837 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6730.g4ba8a190.el8 ceph-noarch 23 k 2023-10-14T11:55:32.837 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mds x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 2.3 M 2023-10-14T11:55:32.837 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6730.g4ba8a190.el8 ceph-noarch 242 k 2023-10-14T11:55:32.837 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mon x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 4.6 M 2023-10-14T11:55:32.838 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-osd x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 18 M 2023-10-14T11:55:32.838 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6730.g4ba8a190.el8 ceph-noarch 15 k 2023-10-14T11:55:32.838 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-selinux x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 24 k 2023-10-14T11:55:32.838 INFO:teuthology.orchestra.run.smithi078.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-10-14T11:55:32.838 INFO:teuthology.orchestra.run.smithi078.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-14T11:55:32.839 INFO:teuthology.orchestra.run.smithi078.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-10-14T11:55:32.839 INFO:teuthology.orchestra.run.smithi078.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-10-14T11:55:32.839 INFO:teuthology.orchestra.run.smithi078.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-10-14T11:55:32.839 INFO:teuthology.orchestra.run.smithi078.stdout: libcephsqlite x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 184 k 2023-10-14T11:55:32.839 INFO:teuthology.orchestra.run.smithi078.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-10-14T11:55:32.839 INFO:teuthology.orchestra.run.smithi078.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-10-14T11:55:32.840 INFO:teuthology.orchestra.run.smithi078.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-14T11:55:32.840 INFO:teuthology.orchestra.run.smithi078.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-10-14T11:55:32.840 INFO:teuthology.orchestra.run.smithi078.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-10-14T11:55:32.840 INFO:teuthology.orchestra.run.smithi078.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-10-14T11:55:32.840 INFO:teuthology.orchestra.run.smithi078.stdout: libradosstriper1 x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 518 k 2023-10-14T11:55:32.841 INFO:teuthology.orchestra.run.smithi078.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-10-14T11:55:32.841 INFO:teuthology.orchestra.run.smithi078.stdout: librgw2 x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 6.9 M 2023-10-14T11:55:32.841 INFO:teuthology.orchestra.run.smithi078.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-10-14T11:55:32.841 INFO:teuthology.orchestra.run.smithi078.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-10-14T11:55:32.841 INFO:teuthology.orchestra.run.smithi078.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-14T11:55:32.842 INFO:teuthology.orchestra.run.smithi078.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-10-14T11:55:32.842 INFO:teuthology.orchestra.run.smithi078.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-10-14T11:55:32.842 INFO:teuthology.orchestra.run.smithi078.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-10-14T11:55:32.842 INFO:teuthology.orchestra.run.smithi078.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-10-14T11:55:32.842 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-10-14T11:55:32.843 INFO:teuthology.orchestra.run.smithi078.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-10-14T11:55:32.843 INFO:teuthology.orchestra.run.smithi078.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-10-14T11:55:32.843 INFO:teuthology.orchestra.run.smithi078.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-10-14T11:55:32.843 INFO:teuthology.orchestra.run.smithi078.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-10-14T11:55:32.844 INFO:teuthology.orchestra.run.smithi078.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-10-14T11:55:32.844 INFO:teuthology.orchestra.run.smithi078.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-14T11:55:32.844 INFO:teuthology.orchestra.run.smithi078.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-14T11:55:32.844 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-14T11:55:32.844 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-argparse x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 45 k 2023-10-14T11:55:32.846 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-common x86_64 2:18.0.0-6730.g4ba8a190.el8 ceph 126 k 2023-10-14T11:55:32.847 INFO:teuthology.orchestra.run.smithi078.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-14T11:55:32.847 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-10-14T11:55:32.847 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-14T11:55:32.847 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-14T11:55:32.847 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-10-14T11:55:32.848 INFO:teuthology.orchestra.run.smithi078.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-14T11:55:32.848 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-14T11:55:32.848 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-14T11:55:32.848 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-10-14T11:55:32.848 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-10-14T11:55:32.849 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-10-14T11:55:32.849 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-14T11:55:32.849 INFO:teuthology.orchestra.run.smithi078.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-14T11:55:32.849 INFO:teuthology.orchestra.run.smithi078.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-14T11:55:32.849 INFO:teuthology.orchestra.run.smithi078.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-10-14T11:55:32.850 INFO:teuthology.orchestra.run.smithi078.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-10-14T11:55:32.850 INFO:teuthology.orchestra.run.smithi078.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-14T11:55:32.850 INFO:teuthology.orchestra.run.smithi078.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-10-14T11:55:32.850 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-10-14T11:55:32.850 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-10-14T11:55:32.851 INFO:teuthology.orchestra.run.smithi078.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-10-14T11:55:32.851 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-14T11:55:32.851 INFO:teuthology.orchestra.run.smithi078.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-14T11:55:32.851 INFO:teuthology.orchestra.run.smithi078.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-10-14T11:55:32.851 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-10-14T11:55:32.852 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-10-14T11:55:32.852 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-10-14T11:55:32.852 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-10-14T11:55:32.852 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-10-14T11:55:32.852 INFO:teuthology.orchestra.run.smithi078.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-14T11:55:32.853 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-14T11:55:32.853 INFO:teuthology.orchestra.run.smithi078.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-14T11:55:32.853 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-14T11:55:32.853 INFO:teuthology.orchestra.run.smithi078.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-10-14T11:55:32.853 INFO:teuthology.orchestra.run.smithi078.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-14T11:55:32.854 INFO:teuthology.orchestra.run.smithi078.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-14T11:55:32.854 INFO:teuthology.orchestra.run.smithi078.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-14T11:55:32.854 INFO:teuthology.orchestra.run.smithi078.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-14T11:55:32.854 INFO:teuthology.orchestra.run.smithi078.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-14T11:55:32.855 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-14T11:55:32.855 INFO:teuthology.orchestra.run.smithi078.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-14T11:55:32.855 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-14T11:55:32.855 INFO:teuthology.orchestra.run.smithi078.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-10-14T11:55:32.856 INFO:teuthology.orchestra.run.smithi078.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-14T11:55:32.856 INFO:teuthology.orchestra.run.smithi078.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-10-14T11:55:32.856 INFO:teuthology.orchestra.run.smithi078.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-10-14T11:55:32.856 INFO:teuthology.orchestra.run.smithi078.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-10-14T11:55:32.856 INFO:teuthology.orchestra.run.smithi078.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-10-14T11:55:32.857 INFO:teuthology.orchestra.run.smithi078.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-10-14T11:55:32.857 INFO:teuthology.orchestra.run.smithi078.stdout:Enabling module streams: 2023-10-14T11:55:32.857 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql 10 2023-10-14T11:55:32.857 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:55:32.857 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2023-10-14T11:55:32.857 INFO:teuthology.orchestra.run.smithi078.stdout:=========================================================================================================================== 2023-10-14T11:55:32.858 INFO:teuthology.orchestra.run.smithi078.stdout:Install 128 Packages 2023-10-14T11:55:32.858 INFO:teuthology.orchestra.run.smithi078.stdout:Upgrade 3 Packages 2023-10-14T11:55:32.858 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:55:32.859 INFO:teuthology.orchestra.run.smithi195.stdout:(2/131): ceph-fuse-18.0.0-6730.g4ba8a190.el8.x8 4.7 MB/s | 918 kB 00:00 2023-10-14T11:55:32.859 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 209 M 2023-10-14T11:55:32.860 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2023-10-14T11:55:32.885 INFO:teuthology.orchestra.run.smithi195.stdout:(3/131): ceph-immutable-object-cache-18.0.0-673 3.3 MB/s | 167 kB 00:00 2023-10-14T11:55:33.111 INFO:teuthology.orchestra.run.smithi195.stdout:(4/131): ceph-mds-18.0.0-6730.g4ba8a190.el8.x86 10 MB/s | 2.3 MB 00:00 2023-10-14T11:55:33.152 INFO:teuthology.orchestra.run.smithi078.stdout:(1/131): ceph-18.0.0-6730.g4ba8a190.el8.x86_64. 24 kB/s | 6.4 kB 00:00 2023-10-14T11:55:33.485 INFO:teuthology.orchestra.run.smithi195.stdout:(5/131): ceph-mgr-18.0.0-6730.g4ba8a190.el8.x86 4.3 MB/s | 1.6 MB 00:00 2023-10-14T11:55:33.611 INFO:teuthology.orchestra.run.smithi195.stdout:(6/131): ceph-base-18.0.0-6730.g4ba8a190.el8.x8 5.0 MB/s | 5.4 MB 00:01 2023-10-14T11:55:33.922 INFO:teuthology.orchestra.run.smithi078.stdout:(2/131): ceph-fuse-18.0.0-6730.g4ba8a190.el8.x8 1.2 MB/s | 918 kB 00:00 2023-10-14T11:55:34.012 INFO:teuthology.orchestra.run.smithi195.stdout:(7/131): ceph-mon-18.0.0-6730.g4ba8a190.el8.x86 8.8 MB/s | 4.6 MB 00:00 2023-10-14T11:55:34.092 INFO:teuthology.orchestra.run.smithi078.stdout:(3/131): ceph-immutable-object-cache-18.0.0-673 970 kB/s | 167 kB 00:00 2023-10-14T11:55:35.753 INFO:teuthology.orchestra.run.smithi078.stdout:(4/131): ceph-base-18.0.0-6730.g4ba8a190.el8.x8 1.9 MB/s | 5.4 MB 00:02 2023-10-14T11:55:35.803 INFO:teuthology.orchestra.run.smithi195.stdout:(8/131): ceph-osd-18.0.0-6730.g4ba8a190.el8.x86 8.2 MB/s | 18 MB 00:02 2023-10-14T11:55:35.945 INFO:teuthology.orchestra.run.smithi195.stdout:(9/131): ceph-radosgw-18.0.0-6730.g4ba8a190.el8 7.2 MB/s | 14 MB 00:01 2023-10-14T11:55:35.971 INFO:teuthology.orchestra.run.smithi195.stdout:(10/131): ceph-selinux-18.0.0-6730.g4ba8a190.el 146 kB/s | 24 kB 00:00 2023-10-14T11:55:35.995 INFO:teuthology.orchestra.run.smithi078.stdout:(5/131): ceph-mgr-18.0.0-6730.g4ba8a190.el8.x86 6.7 MB/s | 1.6 MB 00:00 2023-10-14T11:55:36.046 INFO:teuthology.orchestra.run.smithi195.stdout:(11/131): cephfs-mirror-18.0.0-6730.g4ba8a190.e 3.4 MB/s | 261 kB 00:00 2023-10-14T11:55:36.146 INFO:teuthology.orchestra.run.smithi195.stdout:(12/131): libcephfs-devel-18.0.0-6730.g4ba8a190 310 kB/s | 31 kB 00:00 2023-10-14T11:55:36.245 INFO:teuthology.orchestra.run.smithi078.stdout:(6/131): ceph-mds-18.0.0-6730.g4ba8a190.el8.x86 1.1 MB/s | 2.3 MB 00:02 2023-10-14T11:55:36.321 INFO:teuthology.orchestra.run.smithi195.stdout:(13/131): libcephfs2-18.0.0-6730.g4ba8a190.el8. 4.7 MB/s | 839 kB 00:00 2023-10-14T11:55:36.414 INFO:teuthology.orchestra.run.smithi195.stdout:(14/131): libcephsqlite-18.0.0-6730.g4ba8a190.e 2.0 MB/s | 184 kB 00:00 2023-10-14T11:55:36.506 INFO:teuthology.orchestra.run.smithi195.stdout:(15/131): librados-devel-18.0.0-6730.g4ba8a190. 1.5 MB/s | 139 kB 00:00 2023-10-14T11:55:36.614 INFO:teuthology.orchestra.run.smithi195.stdout:(16/131): libradosstriper1-18.0.0-6730.g4ba8a19 4.7 MB/s | 518 kB 00:00 2023-10-14T11:55:37.071 INFO:teuthology.orchestra.run.smithi078.stdout:(7/131): ceph-common-18.0.0-6730.g4ba8a190.el8. 5.4 MB/s | 23 MB 00:04 2023-10-14T11:55:37.138 INFO:teuthology.orchestra.run.smithi078.stdout:(8/131): ceph-mon-18.0.0-6730.g4ba8a190.el8.x86 4.0 MB/s | 4.6 MB 00:01 2023-10-14T11:55:37.229 INFO:teuthology.orchestra.run.smithi078.stdout:(9/131): ceph-selinux-18.0.0-6730.g4ba8a190.el8 266 kB/s | 24 kB 00:00 2023-10-14T11:55:37.356 INFO:teuthology.orchestra.run.smithi195.stdout:(17/131): librgw2-18.0.0-6730.g4ba8a190.el8.x86 9.3 MB/s | 6.9 MB 00:00 2023-10-14T11:55:37.456 INFO:teuthology.orchestra.run.smithi195.stdout:(18/131): python3-ceph-argparse-18.0.0-6730.g4b 448 kB/s | 45 kB 00:00 2023-10-14T11:55:37.548 INFO:teuthology.orchestra.run.smithi195.stdout:(19/131): python3-ceph-common-18.0.0-6730.g4ba8 1.3 MB/s | 126 kB 00:00 2023-10-14T11:55:37.649 INFO:teuthology.orchestra.run.smithi195.stdout:(20/131): python3-cephfs-18.0.0-6730.g4ba8a190. 1.9 MB/s | 197 kB 00:00 2023-10-14T11:55:37.874 INFO:teuthology.orchestra.run.smithi195.stdout:(21/131): ceph-common-18.0.0-6730.g4ba8a190.el8 4.2 MB/s | 23 MB 00:05 2023-10-14T11:55:37.899 INFO:teuthology.orchestra.run.smithi195.stdout:(22/131): python3-rados-18.0.0-6730.g4ba8a190.e 1.4 MB/s | 371 kB 00:00 2023-10-14T11:55:37.991 INFO:teuthology.orchestra.run.smithi195.stdout:(23/131): python3-rgw-18.0.0-6730.g4ba8a190.el8 1.1 MB/s | 107 kB 00:00 2023-10-14T11:55:38.025 INFO:teuthology.orchestra.run.smithi195.stdout:(24/131): python3-rbd-18.0.0-6730.g4ba8a190.el8 2.4 MB/s | 365 kB 00:00 2023-10-14T11:55:38.075 INFO:teuthology.orchestra.run.smithi195.stdout:(25/131): rbd-fuse-18.0.0-6730.g4ba8a190.el8.x8 1.1 MB/s | 94 kB 00:00 2023-10-14T11:55:38.192 INFO:teuthology.orchestra.run.smithi195.stdout:(26/131): rbd-nbd-18.0.0-6730.g4ba8a190.el8.x86 1.5 MB/s | 179 kB 00:00 2023-10-14T11:55:38.284 INFO:teuthology.orchestra.run.smithi195.stdout:(27/131): ceph-grafana-dashboards-18.0.0-6730.g 252 kB/s | 23 kB 00:00 2023-10-14T11:55:38.376 INFO:teuthology.orchestra.run.smithi195.stdout:(28/131): ceph-mgr-cephadm-18.0.0-6730.g4ba8a19 1.4 MB/s | 132 kB 00:00 2023-10-14T11:55:38.651 INFO:teuthology.orchestra.run.smithi195.stdout:(29/131): ceph-mgr-dashboard-18.0.0-6730.g4ba8a 6.4 MB/s | 1.8 MB 00:00 2023-10-14T11:55:39.094 INFO:teuthology.orchestra.run.smithi195.stdout:(30/131): rbd-mirror-18.0.0-6730.g4ba8a190.el8. 3.4 MB/s | 3.6 MB 00:01 2023-10-14T11:55:39.121 INFO:teuthology.orchestra.run.smithi078.stdout:(10/131): ceph-radosgw-18.0.0-6730.g4ba8a190.el 6.8 MB/s | 14 MB 00:02 2023-10-14T11:55:39.169 INFO:teuthology.orchestra.run.smithi195.stdout:(31/131): ceph-mgr-modules-core-18.0.0-6730.g4b 3.2 MB/s | 242 kB 00:00 2023-10-14T11:55:39.269 INFO:teuthology.orchestra.run.smithi195.stdout:(32/131): ceph-mgr-rook-18.0.0-6730.g4ba8a190.e 474 kB/s | 47 kB 00:00 2023-10-14T11:55:39.288 INFO:teuthology.orchestra.run.smithi078.stdout:(11/131): cephfs-mirror-18.0.0-6730.g4ba8a190.e 1.5 MB/s | 261 kB 00:00 2023-10-14T11:55:39.339 INFO:teuthology.orchestra.run.smithi078.stdout:(12/131): libcephfs-devel-18.0.0-6730.g4ba8a190 620 kB/s | 31 kB 00:00 2023-10-14T11:55:39.411 INFO:teuthology.orchestra.run.smithi195.stdout:(33/131): ceph-mgr-diskprediction-local-18.0.0- 9.8 MB/s | 7.4 MB 00:00 2023-10-14T11:55:39.447 INFO:teuthology.orchestra.run.smithi078.stdout:(13/131): libcephfs2-18.0.0-6730.g4ba8a190.el8. 7.6 MB/s | 839 kB 00:00 2023-10-14T11:55:39.461 INFO:teuthology.orchestra.run.smithi195.stdout:(34/131): ceph-prometheus-alerts-18.0.0-6730.g4 78 kB/s | 15 kB 00:00 2023-10-14T11:55:39.486 INFO:teuthology.orchestra.run.smithi195.stdout:(35/131): ceph-volume-18.0.0-6730.g4ba8a190.el8 3.4 MB/s | 261 kB 00:00 2023-10-14T11:55:39.506 INFO:teuthology.orchestra.run.smithi078.stdout:(14/131): libcephsqlite-18.0.0-6730.g4ba8a190.e 3.1 MB/s | 184 kB 00:00 2023-10-14T11:55:39.537 INFO:teuthology.orchestra.run.smithi195.stdout:(36/131): cephadm-18.0.0-6730.g4ba8a190.el8.noa 2.5 MB/s | 192 kB 00:00 2023-10-14T11:55:39.562 INFO:teuthology.orchestra.run.smithi195.stdout:(37/131): cephfs-top-18.0.0-6730.g4ba8a190.el8. 240 kB/s | 18 kB 00:00 2023-10-14T11:55:39.573 INFO:teuthology.orchestra.run.smithi078.stdout:(15/131): librados-devel-18.0.0-6730.g4ba8a190. 2.0 MB/s | 139 kB 00:00 2023-10-14T11:55:39.657 INFO:teuthology.orchestra.run.smithi078.stdout:(16/131): libradosstriper1-18.0.0-6730.g4ba8a19 6.1 MB/s | 518 kB 00:00 2023-10-14T11:55:39.671 INFO:teuthology.orchestra.run.smithi195.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 543 kB/s | 59 kB 00:00 2023-10-14T11:55:39.721 INFO:teuthology.orchestra.run.smithi195.stdout:(39/131): gperftools-libs-2.7-9.el8.x86_64.rpm 1.6 MB/s | 306 kB 00:00 2023-10-14T11:55:39.746 INFO:teuthology.orchestra.run.smithi195.stdout:(40/131): libunwind-1.3.1-3.el8.x86_64.rpm 1.0 MB/s | 75 kB 00:00 2023-10-14T11:55:39.797 INFO:teuthology.orchestra.run.smithi195.stdout:(41/131): python3-asyncssh-2.7.0-2.el8.noarch.r 6.2 MB/s | 476 kB 00:00 2023-10-14T11:55:39.822 INFO:teuthology.orchestra.run.smithi195.stdout:(42/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 579 kB/s | 44 kB 00:00 2023-10-14T11:55:39.847 INFO:teuthology.orchestra.run.smithi195.stdout:(43/131): python3-beautifulsoup4-4.6.3-2.el8.1. 3.6 MB/s | 185 kB 00:00 2023-10-14T11:55:39.873 INFO:teuthology.orchestra.run.smithi195.stdout:(44/131): python3-cachetools-3.1.1-4.el8.noarch 658 kB/s | 33 kB 00:00 2023-10-14T11:55:39.898 INFO:teuthology.orchestra.run.smithi195.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 325 kB/s | 16 kB 00:00 2023-10-14T11:55:39.948 INFO:teuthology.orchestra.run.smithi195.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 2.2 MB/s | 173 kB 00:00 2023-10-14T11:55:39.973 INFO:teuthology.orchestra.run.smithi195.stdout:(47/131): python3-cherrypy-18.4.0-1.el8.noarch. 5.0 MB/s | 384 kB 00:00 2023-10-14T11:55:39.999 INFO:teuthology.orchestra.run.smithi195.stdout:(48/131): python3-google-auth-1.1.1-10.el8.noar 1.6 MB/s | 81 kB 00:00 2023-10-14T11:55:40.016 INFO:teuthology.orchestra.run.smithi195.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 265 kB/s | 11 kB 00:00 2023-10-14T11:55:40.041 INFO:teuthology.orchestra.run.smithi195.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 420 kB/s | 18 kB 00:00 2023-10-14T11:55:40.100 INFO:teuthology.orchestra.run.smithi195.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 836 kB/s | 49 kB 00:00 2023-10-14T11:55:40.134 INFO:teuthology.orchestra.run.smithi195.stdout:(52/131): python3-kubernetes-11.0.0-6.el8.noarc 9.9 MB/s | 1.2 MB 00:00 2023-10-14T11:55:40.150 INFO:teuthology.orchestra.run.smithi195.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 1.1 MB/s | 59 kB 00:00 2023-10-14T11:55:40.167 INFO:teuthology.orchestra.run.smithi195.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 1.8 MB/s | 60 kB 00:00 2023-10-14T11:55:40.201 INFO:teuthology.orchestra.run.smithi195.stdout:(55/131): python3-pecan-1.3.2-9.el8.noarch.rpm 5.5 MB/s | 283 kB 00:00 2023-10-14T11:55:40.218 INFO:teuthology.orchestra.run.smithi195.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 309 kB/s | 16 kB 00:00 2023-10-14T11:55:40.235 INFO:teuthology.orchestra.run.smithi195.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 1.0 MB/s | 34 kB 00:00 2023-10-14T11:55:40.269 INFO:teuthology.orchestra.run.smithi195.stdout:(58/131): python3-routes-2.4.1-12.el8.noarch.rp 3.8 MB/s | 196 kB 00:00 2023-10-14T11:55:40.286 INFO:teuthology.orchestra.run.smithi195.stdout:(59/131): python3-rsa-4.9-2.el8.noarch.rpm 1.2 MB/s | 64 kB 00:00 2023-10-14T11:55:40.302 INFO:teuthology.orchestra.run.smithi195.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 557 kB/s | 19 kB 00:00 2023-10-14T11:55:40.328 INFO:teuthology.orchestra.run.smithi195.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 573 kB/s | 24 kB 00:00 2023-10-14T11:55:40.353 INFO:teuthology.orchestra.run.smithi195.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 582 kB/s | 29 kB 00:00 2023-10-14T11:55:40.378 INFO:teuthology.orchestra.run.smithi195.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 548 kB/s | 27 kB 00:00 2023-10-14T11:55:40.398 INFO:teuthology.orchestra.run.smithi078.stdout:(17/131): librgw2-18.0.0-6730.g4ba8a190.el8.x86 9.3 MB/s | 6.9 MB 00:00 2023-10-14T11:55:40.404 INFO:teuthology.orchestra.run.smithi195.stdout:(64/131): python3-waitress-1.4.3-1.el8.noarch.r 5.0 MB/s | 255 kB 00:00 2023-10-14T11:55:40.429 INFO:teuthology.orchestra.run.smithi195.stdout:(65/131): python3-webob-1.8.5-1.el8.1.noarch.rp 4.9 MB/s | 251 kB 00:00 2023-10-14T11:55:40.449 INFO:teuthology.orchestra.run.smithi078.stdout:(18/131): python3-ceph-argparse-18.0.0-6730.g4b 895 kB/s | 45 kB 00:00 2023-10-14T11:55:40.454 INFO:teuthology.orchestra.run.smithi195.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 1.2 MB/s | 61 kB 00:00 2023-10-14T11:55:40.480 INFO:teuthology.orchestra.run.smithi195.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 1.7 MB/s | 85 kB 00:00 2023-10-14T11:55:40.499 INFO:teuthology.orchestra.run.smithi078.stdout:(19/131): python3-ceph-common-18.0.0-6730.g4ba8 2.5 MB/s | 126 kB 00:00 2023-10-14T11:55:40.505 INFO:teuthology.orchestra.run.smithi195.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 458 kB/s | 23 kB 00:00 2023-10-14T11:55:40.530 INFO:teuthology.orchestra.run.smithi195.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.8 MB/s | 70 kB 00:00 2023-10-14T11:55:40.589 INFO:teuthology.orchestra.run.smithi195.stdout:(70/131): thrift-0.13.0-2.el8.x86_64.rpm 16 MB/s | 1.7 MB 00:00 2023-10-14T11:55:40.591 INFO:teuthology.orchestra.run.smithi078.stdout:(20/131): python3-cephfs-18.0.0-6730.g4ba8a190. 2.1 MB/s | 197 kB 00:00 2023-10-14T11:55:40.650 INFO:teuthology.orchestra.run.smithi078.stdout:(21/131): python3-rados-18.0.0-6730.g4ba8a190.e 6.2 MB/s | 371 kB 00:00 2023-10-14T11:55:40.655 INFO:teuthology.orchestra.run.smithi195.stdout:(71/131): python3-pyasn1-modules-0.3.7-6.el8.no 1.6 MB/s | 110 kB 00:00 2023-10-14T11:55:40.698 INFO:teuthology.orchestra.run.smithi195.stdout:(72/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.3 MB/s | 54 kB 00:00 2023-10-14T11:55:40.700 INFO:teuthology.orchestra.run.smithi078.stdout:(22/131): python3-rbd-18.0.0-6730.g4ba8a190.el8 7.2 MB/s | 365 kB 00:00 2023-10-14T11:55:40.748 INFO:teuthology.orchestra.run.smithi195.stdout:(73/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 5.1 MB/s | 259 kB 00:00 2023-10-14T11:55:40.750 INFO:teuthology.orchestra.run.smithi078.stdout:(23/131): python3-rgw-18.0.0-6730.g4ba8a190.el8 2.1 MB/s | 107 kB 00:00 2023-10-14T11:55:40.798 INFO:teuthology.orchestra.run.smithi195.stdout:(74/131): socat-1.7.4.1-1.el8.x86_64.rpm 6.3 MB/s | 323 kB 00:00 2023-10-14T11:55:40.801 INFO:teuthology.orchestra.run.smithi078.stdout:(24/131): rbd-fuse-18.0.0-6730.g4ba8a190.el8.x8 1.8 MB/s | 94 kB 00:00 2023-10-14T11:55:40.840 INFO:teuthology.orchestra.run.smithi195.stdout:(75/131): python3-markupsafe-0.23-19.el8.x86_64 937 kB/s | 39 kB 00:00 2023-10-14T11:55:40.891 INFO:teuthology.orchestra.run.smithi195.stdout:(76/131): bison-3.0.4-10.el8.x86_64.rpm 13 MB/s | 688 kB 00:00 2023-10-14T11:55:41.032 INFO:teuthology.orchestra.run.smithi195.stdout:(77/131): python3-scipy-1.0.0-21.module+el8.5.0 27 MB/s | 14 MB 00:00 2023-10-14T11:55:41.074 INFO:teuthology.orchestra.run.smithi195.stdout:(78/131): python3-jinja2-2.10.1-3.el8.noarch.rp 2.9 MB/s | 538 kB 00:00 2023-10-14T11:55:41.100 INFO:teuthology.orchestra.run.smithi195.stdout:(79/131): python3-mako-1.0.6-13.el8.noarch.rpm 2.3 MB/s | 157 kB 00:00 2023-10-14T11:55:41.125 INFO:teuthology.orchestra.run.smithi195.stdout:(80/131): oniguruma-6.8.2-2.el8.x86_64.rpm 3.6 MB/s | 187 kB 00:00 2023-10-14T11:55:41.159 INFO:teuthology.orchestra.run.smithi195.stdout:(81/131): automake-1.16.1-7.el8.noarch.rpm 12 MB/s | 713 kB 00:00 2023-10-14T11:55:41.201 INFO:teuthology.orchestra.run.smithi195.stdout:(82/131): python3-jsonpatch-1.21-2.el8.noarch.r 636 kB/s | 27 kB 00:00 2023-10-14T11:55:41.210 INFO:teuthology.orchestra.run.smithi078.stdout:(25/131): rbd-mirror-18.0.0-6730.g4ba8a190.el8. 8.8 MB/s | 3.6 MB 00:00 2023-10-14T11:55:41.251 INFO:teuthology.orchestra.run.smithi195.stdout:(83/131): flex-2.6.1-9.el8.x86_64.rpm 6.2 MB/s | 320 kB 00:00 2023-10-14T11:55:41.268 INFO:teuthology.orchestra.run.smithi078.stdout:(26/131): rbd-nbd-18.0.0-6730.g4ba8a190.el8.x86 3.0 MB/s | 179 kB 00:00 2023-10-14T11:55:41.318 INFO:teuthology.orchestra.run.smithi195.stdout:(84/131): python3-babel-2.5.1-7.el8.noarch.rpm 25 MB/s | 4.8 MB 00:00 2023-10-14T11:55:41.327 INFO:teuthology.orchestra.run.smithi078.stdout:(27/131): ceph-grafana-dashboards-18.0.0-6730.g 395 kB/s | 23 kB 00:00 2023-10-14T11:55:41.343 INFO:teuthology.orchestra.run.smithi195.stdout:(85/131): python3-prettytable-0.7.2-14.el8.noar 478 kB/s | 44 kB 00:00 2023-10-14T11:55:41.385 INFO:teuthology.orchestra.run.smithi078.stdout:(28/131): ceph-mgr-cephadm-18.0.0-6730.g4ba8a19 2.2 MB/s | 132 kB 00:00 2023-10-14T11:55:41.402 INFO:teuthology.orchestra.run.smithi195.stdout:(86/131): python3-werkzeug-0.12.2-4.el8.noarch. 7.7 MB/s | 457 kB 00:00 2023-10-14T11:55:41.461 INFO:teuthology.orchestra.run.smithi195.stdout:(87/131): python3-numpy-1.14.3-10.el8.x86_64.rp 26 MB/s | 3.7 MB 00:00 2023-10-14T11:55:41.486 INFO:teuthology.orchestra.run.smithi195.stdout:(88/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 2.6 MB/s | 225 kB 00:00 2023-10-14T11:55:41.544 INFO:teuthology.orchestra.run.smithi195.stdout:(89/131): uuid-1.6.2-43.el8.x86_64.rpm 767 kB/s | 64 kB 00:00 2023-10-14T11:55:41.586 INFO:teuthology.orchestra.run.smithi195.stdout:(90/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.2 MB/s | 126 kB 00:00 2023-10-14T11:55:41.611 INFO:teuthology.orchestra.run.smithi078.stdout:(29/131): ceph-mgr-dashboard-18.0.0-6730.g4ba8a 7.8 MB/s | 1.8 MB 00:00 2023-10-14T11:55:41.612 INFO:teuthology.orchestra.run.smithi195.stdout:(91/131): python3-jsonpointer-1.10-11.el8.noarc 298 kB/s | 20 kB 00:00 2023-10-14T11:55:41.637 INFO:teuthology.orchestra.run.smithi195.stdout:(92/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 2.0 MB/s | 103 kB 00:00 2023-10-14T11:55:41.670 INFO:teuthology.orchestra.run.smithi195.stdout:(93/131): libtool-2.4.6-25.el8.x86_64.rpm 12 MB/s | 709 kB 00:00 2023-10-14T11:55:41.721 INFO:teuthology.orchestra.run.smithi195.stdout:(94/131): libpq-13.5-1.el8.x86_64.rpm 3.9 MB/s | 198 kB 00:00 2023-10-14T11:55:41.796 INFO:teuthology.orchestra.run.smithi195.stdout:(95/131): openblas-threads-0.3.15-3.el8.x86_64. 31 MB/s | 4.9 MB 00:00 2023-10-14T11:55:41.872 INFO:teuthology.orchestra.run.smithi195.stdout:(96/131): jq-1.6-3.el8.x86_64.rpm 2.6 MB/s | 202 kB 00:00 2023-10-14T11:55:41.930 INFO:teuthology.orchestra.run.smithi195.stdout:(97/131): librdkafka-0.11.4-3.el8.x86_64.rpm 5.9 MB/s | 354 kB 00:00 2023-10-14T11:55:42.030 INFO:teuthology.orchestra.run.smithi195.stdout:(98/131): openblas-0.3.15-3.el8.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2023-10-14T11:55:42.148 INFO:teuthology.orchestra.run.smithi195.stdout:(99/131): postgresql-server-10.21-2.module+el8. 23 MB/s | 5.1 MB 00:00 2023-10-14T11:55:42.181 INFO:teuthology.orchestra.run.smithi195.stdout:(100/131): postgresql-contrib-10.21-2.module+el 5.3 MB/s | 808 kB 00:00 2023-10-14T11:55:42.223 INFO:teuthology.orchestra.run.smithi195.stdout:(101/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2023-10-14T11:55:42.394 INFO:teuthology.orchestra.run.smithi078.stdout:(30/131): ceph-mgr-diskprediction-local-18.0.0- 9.5 MB/s | 7.4 MB 00:00 2023-10-14T11:55:42.453 INFO:teuthology.orchestra.run.smithi078.stdout:(31/131): ceph-mgr-modules-core-18.0.0-6730.g4b 4.1 MB/s | 242 kB 00:00 2023-10-14T11:55:42.503 INFO:teuthology.orchestra.run.smithi078.stdout:(32/131): ceph-mgr-rook-18.0.0-6730.g4ba8a190.e 951 kB/s | 47 kB 00:00 2023-10-14T11:55:42.553 INFO:teuthology.orchestra.run.smithi078.stdout:(33/131): ceph-prometheus-alerts-18.0.0-6730.g4 300 kB/s | 15 kB 00:00 2023-10-14T11:55:42.629 INFO:teuthology.orchestra.run.smithi078.stdout:(34/131): ceph-volume-18.0.0-6730.g4ba8a190.el8 3.4 MB/s | 261 kB 00:00 2023-10-14T11:55:42.682 INFO:teuthology.orchestra.run.smithi195.stdout:(102/131): ceph-test-18.0.0-6730.g4ba8a190.el8. 7.3 MB/s | 49 MB 00:06 2023-10-14T11:55:42.687 INFO:teuthology.orchestra.run.smithi078.stdout:(35/131): cephadm-18.0.0-6730.g4ba8a190.el8.noa 3.2 MB/s | 192 kB 00:00 2023-10-14T11:55:42.715 INFO:teuthology.orchestra.run.smithi195.stdout:(103/131): postgresql-10.21-2.module+el8.6.0+15 2.6 MB/s | 1.5 MB 00:00 2023-10-14T11:55:42.730 INFO:teuthology.orchestra.run.smithi078.stdout:(36/131): cephfs-top-18.0.0-6730.g4ba8a190.el8. 433 kB/s | 18 kB 00:00 2023-10-14T11:55:42.749 INFO:teuthology.orchestra.run.smithi195.stdout:(104/131): libselinux-devel-2.9-5.el8.x86_64.rp 380 kB/s | 200 kB 00:00 2023-10-14T11:55:42.774 INFO:teuthology.orchestra.run.smithi195.stdout:(105/131): python3-jwt-1.6.1-2.el8.noarch.rpm 464 kB/s | 43 kB 00:00 2023-10-14T11:55:42.799 INFO:teuthology.orchestra.run.smithi195.stdout:(106/131): keyutils-libs-devel-1.5.10-9.el8.x86 962 kB/s | 48 kB 00:00 2023-10-14T11:55:42.825 INFO:teuthology.orchestra.run.smithi195.stdout:(107/131): libsepol-devel-2.9-3.el8.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2023-10-14T11:55:42.850 INFO:teuthology.orchestra.run.smithi195.stdout:(108/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.7 MB/s | 220 kB 00:00 2023-10-14T11:55:42.875 INFO:teuthology.orchestra.run.smithi195.stdout:(109/131): python3-requests-oauthlib-1.0.0-1.el 564 kB/s | 43 kB 00:00 2023-10-14T11:55:42.901 INFO:teuthology.orchestra.run.smithi195.stdout:(110/131): python3-pycparser-2.14-14.el8.noarch 1.4 MB/s | 109 kB 00:00 2023-10-14T11:55:42.926 INFO:teuthology.orchestra.run.smithi195.stdout:(111/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 3.0 MB/s | 229 kB 00:00 2023-10-14T11:55:42.951 INFO:teuthology.orchestra.run.smithi195.stdout:(112/131): libaio-devel-0.3.112-1.el8.x86_64.rp 372 kB/s | 19 kB 00:00 2023-10-14T11:55:42.976 INFO:teuthology.orchestra.run.smithi195.stdout:(113/131): python3-oauthlib-2.1.0-1.el8.noarch. 3.0 MB/s | 155 kB 00:00 2023-10-14T11:55:43.002 INFO:teuthology.orchestra.run.smithi195.stdout:(114/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 4.3 MB/s | 560 kB 00:00 2023-10-14T11:55:43.019 INFO:teuthology.orchestra.run.smithi195.stdout:(115/131): libverto-devel-0.3.0-5.el8.x86_64.rp 427 kB/s | 18 kB 00:00 2023-10-14T11:55:43.046 INFO:teuthology.orchestra.run.smithi078.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 967 kB/s | 306 kB 00:00 2023-10-14T11:55:43.052 INFO:teuthology.orchestra.run.smithi195.stdout:(116/131): python3-cryptography-3.2.1-5.el8.x86 5.4 MB/s | 559 kB 00:00 2023-10-14T11:55:43.078 INFO:teuthology.orchestra.run.smithi195.stdout:(117/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 4.0 MB/s | 238 kB 00:00 2023-10-14T11:55:43.105 INFO:teuthology.orchestra.run.smithi078.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 1.0 MB/s | 59 kB 00:00 2023-10-14T11:55:43.120 INFO:teuthology.orchestra.run.smithi195.stdout:(118/131): pcre2-devel-10.32-2.el8.x86_64.rpm 5.0 MB/s | 605 kB 00:00 2023-10-14T11:55:43.145 INFO:teuthology.orchestra.run.smithi195.stdout:(119/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 1.1 MB/s | 72 kB 00:00 2023-10-14T11:55:43.170 INFO:teuthology.orchestra.run.smithi195.stdout:(120/131): fuse-2.9.7-15.el8.x86_64.rpm 1.6 MB/s | 83 kB 00:00 2023-10-14T11:55:43.172 INFO:teuthology.orchestra.run.smithi078.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 1.1 MB/s | 75 kB 00:00 2023-10-14T11:55:43.187 INFO:teuthology.orchestra.run.smithi195.stdout:(121/131): elfutils-libelf-devel-0.186-1.el8.x8 1.4 MB/s | 60 kB 00:00 2023-10-14T11:55:43.221 INFO:teuthology.orchestra.run.smithi195.stdout:(122/131): libuuid-devel-2.32.1-35.el8.x86_64.r 1.9 MB/s | 98 kB 00:00 2023-10-14T11:55:43.238 INFO:teuthology.orchestra.run.smithi195.stdout:(123/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 1.1 MB/s | 58 kB 00:00 2023-10-14T11:55:43.289 INFO:teuthology.orchestra.run.smithi078.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 4.0 MB/s | 476 kB 00:00 2023-10-14T11:55:43.321 INFO:teuthology.orchestra.run.smithi195.stdout:(124/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 8.7 MB/s | 2.3 MB 00:00 2023-10-14T11:55:43.340 INFO:teuthology.orchestra.run.smithi195.stdout:(125/131): libcom_err-devel-1.45.6-4.el8.x86_64 329 kB/s | 39 kB 00:00 2023-10-14T11:55:43.349 INFO:teuthology.orchestra.run.smithi078.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 748 kB/s | 44 kB 00:00 2023-10-14T11:55:43.363 INFO:teuthology.orchestra.run.smithi195.stdout:(126/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 4.0 MB/s | 171 kB 00:00 2023-10-14T11:55:43.389 INFO:teuthology.orchestra.run.smithi195.stdout:(127/131): NetworkManager-initscripts-updown-1. 2.7 MB/s | 138 kB 00:00 2023-10-14T11:55:43.415 INFO:teuthology.orchestra.run.smithi195.stdout:(128/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 3.6 MB/s | 644 kB 00:00 2023-10-14T11:55:43.415 INFO:teuthology.orchestra.run.smithi078.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 2.7 MB/s | 185 kB 00:00 2023-10-14T11:55:43.481 INFO:teuthology.orchestra.run.smithi195.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 12 MB/s | 798 kB 00:00 2023-10-14T11:55:43.731 INFO:teuthology.orchestra.run.smithi195.stdout:(130/131): librados2-18.0.0-6730.g4ba8a190.el8. 10 MB/s | 3.7 MB 00:00 2023-10-14T11:55:43.865 INFO:teuthology.orchestra.run.smithi078.stdout:(43/131): ceph-test-18.0.0-6730.g4ba8a190.el8.x 7.4 MB/s | 49 MB 00:06 2023-10-14T11:55:43.873 INFO:teuthology.orchestra.run.smithi195.stdout:(131/131): librbd1-18.0.0-6730.g4ba8a190.el8.x8 7.7 MB/s | 3.7 MB 00:00 2023-10-14T11:55:43.876 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2023-10-14T11:55:43.876 INFO:teuthology.orchestra.run.smithi195.stdout:Total 18 MB/s | 209 MB 00:11 2023-10-14T11:55:43.890 INFO:teuthology.orchestra.run.smithi078.stdout:(44/131): python3-cachetools-3.1.1-4.el8.noarch 69 kB/s | 33 kB 00:00 2023-10-14T11:55:44.074 INFO:teuthology.orchestra.run.smithi078.stdout:(45/131): ceph-osd-18.0.0-6730.g4ba8a190.el8.x8 2.3 MB/s | 18 MB 00:07 2023-10-14T11:55:44.100 INFO:teuthology.orchestra.run.smithi078.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 827 kB/s | 173 kB 00:00 2023-10-14T11:55:44.175 INFO:teuthology.orchestra.run.smithi078.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 1.1 MB/s | 81 kB 00:00 2023-10-14T11:55:44.200 INFO:teuthology.orchestra.run.smithi078.stdout:(48/131): python3-certifi-2018.10.15-7.el8.noar 49 kB/s | 16 kB 00:00 2023-10-14T11:55:44.242 INFO:teuthology.orchestra.run.smithi078.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 166 kB/s | 11 kB 00:00 2023-10-14T11:55:44.267 INFO:teuthology.orchestra.run.smithi078.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 262 kB/s | 18 kB 00:00 2023-10-14T11:55:44.559 INFO:teuthology.orchestra.run.smithi078.stdout:(51/131): python3-cherrypy-18.4.0-1.el8.noarch. 794 kB/s | 384 kB 00:00 2023-10-14T11:55:44.593 INFO:teuthology.orchestra.run.smithi078.stdout:(52/131): python3-logutils-0.3.5-11.el8.noarch. 150 kB/s | 49 kB 00:00 2023-10-14T11:55:44.627 INFO:teuthology.orchestra.run.smithi078.stdout:(53/131): python3-kubernetes-11.0.0-6.el8.noarc 3.0 MB/s | 1.2 MB 00:00 2023-10-14T11:55:44.652 INFO:teuthology.orchestra.run.smithi078.stdout:(54/131): python3-more-itertools-7.2.0-3.el8.no 631 kB/s | 59 kB 00:00 2023-10-14T11:55:44.677 INFO:teuthology.orchestra.run.smithi078.stdout:(55/131): python3-natsort-7.1.1-2.el8.noarch.rp 718 kB/s | 60 kB 00:00 2023-10-14T11:55:44.688 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2023-10-14T11:55:44.702 INFO:teuthology.orchestra.run.smithi078.stdout:(56/131): python3-pecan-1.3.2-9.el8.noarch.rpm 3.7 MB/s | 283 kB 00:00 2023-10-14T11:55:44.728 INFO:teuthology.orchestra.run.smithi078.stdout:(57/131): python3-portend-2.6-1.el8.noarch.rpm 206 kB/s | 16 kB 00:00 2023-10-14T11:55:44.753 INFO:teuthology.orchestra.run.smithi078.stdout:(58/131): python3-repoze-lru-0.7-6.el8.noarch.r 445 kB/s | 34 kB 00:00 2023-10-14T11:55:44.779 INFO:teuthology.orchestra.run.smithi078.stdout:(59/131): python3-routes-2.4.1-12.el8.noarch.rp 2.5 MB/s | 196 kB 00:00 2023-10-14T11:55:44.796 INFO:teuthology.orchestra.run.smithi078.stdout:(60/131): python3-rsa-4.9-2.el8.noarch.rpm 956 kB/s | 64 kB 00:00 2023-10-14T11:55:44.807 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2023-10-14T11:55:44.807 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2023-10-14T11:55:44.821 INFO:teuthology.orchestra.run.smithi078.stdout:(61/131): python3-simplegeneric-0.8.1-17.el8.no 279 kB/s | 19 kB 00:00 2023-10-14T11:55:44.846 INFO:teuthology.orchestra.run.smithi078.stdout:(62/131): python3-singledispatch-3.4.0.3-18.el8 355 kB/s | 24 kB 00:00 2023-10-14T11:55:44.897 INFO:teuthology.orchestra.run.smithi078.stdout:(63/131): python3-tempora-1.14.1-5.el8.noarch.r 293 kB/s | 29 kB 00:00 2023-10-14T11:55:44.930 INFO:teuthology.orchestra.run.smithi078.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 253 kB/s | 27 kB 00:00 2023-10-14T11:55:44.955 INFO:teuthology.orchestra.run.smithi078.stdout:(65/131): python3-waitress-1.4.3-1.el8.noarch.r 2.3 MB/s | 255 kB 00:00 2023-10-14T11:55:44.989 INFO:teuthology.orchestra.run.smithi078.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 1.0 MB/s | 61 kB 00:00 2023-10-14T11:55:45.023 INFO:teuthology.orchestra.run.smithi078.stdout:(67/131): python3-webob-1.8.5-1.el8.1.noarch.rp 1.9 MB/s | 251 kB 00:00 2023-10-14T11:55:45.056 INFO:teuthology.orchestra.run.smithi078.stdout:(68/131): python3-webtest-2.0.33-1.el8.noarch.r 849 kB/s | 85 kB 00:00 2023-10-14T11:55:45.073 INFO:teuthology.orchestra.run.smithi078.stdout:(69/131): python3-zc-lockfile-2.0-2.el8.noarch. 273 kB/s | 23 kB 00:00 2023-10-14T11:55:45.098 INFO:teuthology.orchestra.run.smithi078.stdout:(70/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.6 MB/s | 70 kB 00:00 2023-10-14T11:55:45.182 INFO:teuthology.orchestra.run.smithi078.stdout:(71/131): thrift-0.13.0-2.el8.x86_64.rpm 11 MB/s | 1.7 MB 00:00 2023-10-14T11:55:45.207 INFO:teuthology.orchestra.run.smithi078.stdout:(72/131): python3-pyasn1-modules-0.3.7-6.el8.no 1.0 MB/s | 110 kB 00:00 2023-10-14T11:55:45.258 INFO:teuthology.orchestra.run.smithi078.stdout:(73/131): python3-pytz-2017.2-9.el8.noarch.rpm 717 kB/s | 54 kB 00:00 2023-10-14T11:55:45.283 INFO:teuthology.orchestra.run.smithi078.stdout:(74/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 3.6 MB/s | 259 kB 00:00 2023-10-14T11:55:45.317 INFO:teuthology.orchestra.run.smithi078.stdout:(75/131): socat-1.7.4.1-1.el8.x86_64.rpm 5.4 MB/s | 323 kB 00:00 2023-10-14T11:55:45.342 INFO:teuthology.orchestra.run.smithi078.stdout:(76/131): python3-markupsafe-0.23-19.el8.x86_64 664 kB/s | 39 kB 00:00 2023-10-14T11:55:45.376 INFO:teuthology.orchestra.run.smithi078.stdout:(77/131): bison-3.0.4-10.el8.x86_64.rpm 12 MB/s | 688 kB 00:00 2023-10-14T11:55:45.409 INFO:teuthology.orchestra.run.smithi078.stdout:(78/131): python3-jinja2-2.10.1-3.el8.noarch.rp 7.9 MB/s | 538 kB 00:00 2023-10-14T11:55:45.426 INFO:teuthology.orchestra.run.smithi078.stdout:(79/131): python3-mako-1.0.6-13.el8.noarch.rpm 3.0 MB/s | 157 kB 00:00 2023-10-14T11:55:45.451 INFO:teuthology.orchestra.run.smithi078.stdout:(80/131): oniguruma-6.8.2-2.el8.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2023-10-14T11:55:45.494 INFO:teuthology.orchestra.run.smithi078.stdout:(81/131): automake-1.16.1-7.el8.noarch.rpm 10 MB/s | 713 kB 00:00 2023-10-14T11:55:45.535 INFO:teuthology.orchestra.run.smithi078.stdout:(82/131): python3-jsonpatch-1.21-2.el8.noarch.r 640 kB/s | 27 kB 00:00 2023-10-14T11:55:45.602 INFO:teuthology.orchestra.run.smithi078.stdout:(83/131): flex-2.6.1-9.el8.x86_64.rpm 4.7 MB/s | 320 kB 00:00 2023-10-14T11:55:45.669 INFO:teuthology.orchestra.run.smithi078.stdout:(84/131): python3-babel-2.5.1-7.el8.noarch.rpm 22 MB/s | 4.8 MB 00:00 2023-10-14T11:55:45.695 INFO:teuthology.orchestra.run.smithi078.stdout:(85/131): python3-prettytable-0.7.2-14.el8.noar 479 kB/s | 44 kB 00:00 2023-10-14T11:55:45.745 INFO:teuthology.orchestra.run.smithi078.stdout:(86/131): python3-werkzeug-0.12.2-4.el8.noarch. 8.9 MB/s | 457 kB 00:00 2023-10-14T11:55:45.923 INFO:teuthology.orchestra.run.smithi078.stdout:(87/131): python3-scipy-1.0.0-21.module+el8.5.0 16 MB/s | 14 MB 00:00 2023-10-14T11:55:45.928 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2023-10-14T11:55:45.932 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2023-10-14T11:55:45.954 INFO:teuthology.orchestra.run.smithi078.stdout:(88/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.1 MB/s | 225 kB 00:00 2023-10-14T11:55:45.979 INFO:teuthology.orchestra.run.smithi078.stdout:(89/131): uuid-1.6.2-43.el8.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2023-10-14T11:55:46.004 INFO:teuthology.orchestra.run.smithi078.stdout:(90/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.5 MB/s | 126 kB 00:00 2023-10-14T11:55:46.063 INFO:teuthology.orchestra.run.smithi078.stdout:(91/131): python3-numpy-1.14.3-10.el8.x86_64.rp 9.5 MB/s | 3.7 MB 00:00 2023-10-14T11:55:46.088 INFO:teuthology.orchestra.run.smithi078.stdout:(92/131): python3-jsonpointer-1.10-11.el8.noarc 183 kB/s | 20 kB 00:00 2023-10-14T11:55:46.113 INFO:teuthology.orchestra.run.smithi078.stdout:(93/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 951 kB/s | 103 kB 00:00 2023-10-14T11:55:46.181 INFO:teuthology.orchestra.run.smithi078.stdout:(94/131): libtool-2.4.6-25.el8.x86_64.rpm 7.5 MB/s | 709 kB 00:00 2023-10-14T11:55:46.206 INFO:teuthology.orchestra.run.smithi078.stdout:(95/131): libpq-13.5-1.el8.x86_64.rpm 2.1 MB/s | 198 kB 00:00 2023-10-14T11:55:46.289 INFO:teuthology.orchestra.run.smithi078.stdout:(96/131): jq-1.6-3.el8.x86_64.rpm 2.4 MB/s | 202 kB 00:00 2023-10-14T11:55:46.340 INFO:teuthology.orchestra.run.smithi078.stdout:(97/131): librdkafka-0.11.4-3.el8.x86_64.rpm 6.9 MB/s | 354 kB 00:00 2023-10-14T11:55:46.423 INFO:teuthology.orchestra.run.smithi078.stdout:(98/131): openblas-threads-0.3.15-3.el8.x86_64. 15 MB/s | 4.9 MB 00:00 2023-10-14T11:55:46.490 INFO:teuthology.orchestra.run.smithi078.stdout:(99/131): openblas-0.3.15-3.el8.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2023-10-14T11:55:46.532 INFO:teuthology.orchestra.run.smithi078.stdout:(100/131): postgresql-contrib-10.21-2.module+el 7.3 MB/s | 808 kB 00:00 2023-10-14T11:55:46.605 INFO:teuthology.orchestra.run.smithi078.stdout:(101/131): postgresql-10.21-2.module+el8.6.0+15 13 MB/s | 1.5 MB 00:00 2023-10-14T11:55:46.633 INFO:teuthology.orchestra.run.smithi078.stdout:(102/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 466 kB/s | 47 kB 00:00 2023-10-14T11:55:46.658 INFO:teuthology.orchestra.run.smithi078.stdout:(103/131): libselinux-devel-2.9-5.el8.x86_64.rp 3.7 MB/s | 200 kB 00:00 2023-10-14T11:55:46.684 INFO:teuthology.orchestra.run.smithi078.stdout:(104/131): python3-jwt-1.6.1-2.el8.noarch.rpm 847 kB/s | 43 kB 00:00 2023-10-14T11:55:46.701 INFO:teuthology.orchestra.run.smithi078.stdout:(105/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 5.1 MB/s | 220 kB 00:00 2023-10-14T11:55:46.726 INFO:teuthology.orchestra.run.smithi078.stdout:(106/131): keyutils-libs-devel-1.5.10-9.el8.x86 1.1 MB/s | 48 kB 00:00 2023-10-14T11:55:46.751 INFO:teuthology.orchestra.run.smithi078.stdout:(107/131): libsepol-devel-2.9-3.el8.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2023-10-14T11:55:46.776 INFO:teuthology.orchestra.run.smithi078.stdout:(108/131): python3-requests-oauthlib-1.0.0-1.el 851 kB/s | 43 kB 00:00 2023-10-14T11:55:46.802 INFO:teuthology.orchestra.run.smithi078.stdout:(109/131): python3-pycparser-2.14-14.el8.noarch 2.1 MB/s | 109 kB 00:00 2023-10-14T11:55:46.827 INFO:teuthology.orchestra.run.smithi078.stdout:(110/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 4.4 MB/s | 229 kB 00:00 2023-10-14T11:55:46.869 INFO:teuthology.orchestra.run.smithi078.stdout:(111/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 8.2 MB/s | 560 kB 00:00 2023-10-14T11:55:46.895 INFO:teuthology.orchestra.run.smithi078.stdout:(112/131): libaio-devel-0.3.112-1.el8.x86_64.rp 279 kB/s | 19 kB 00:00 2023-10-14T11:55:46.920 INFO:teuthology.orchestra.run.smithi078.stdout:(113/131): python3-oauthlib-2.1.0-1.el8.noarch. 3.0 MB/s | 155 kB 00:00 2023-10-14T11:55:46.978 INFO:teuthology.orchestra.run.smithi078.stdout:(114/131): python3-cryptography-3.2.1-5.el8.x86 6.5 MB/s | 559 kB 00:00 2023-10-14T11:55:47.004 INFO:teuthology.orchestra.run.smithi078.stdout:(115/131): libverto-devel-0.3.0-5.el8.x86_64.rp 214 kB/s | 18 kB 00:00 2023-10-14T11:55:47.071 INFO:teuthology.orchestra.run.smithi078.stdout:(116/131): postgresql-server-10.21-2.module+el8 6.9 MB/s | 5.1 MB 00:00 2023-10-14T11:55:47.096 INFO:teuthology.orchestra.run.smithi078.stdout:(117/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 2.5 MB/s | 238 kB 00:00 2023-10-14T11:55:47.138 INFO:teuthology.orchestra.run.smithi078.stdout:(118/131): pcre2-devel-10.32-2.el8.x86_64.rpm 3.7 MB/s | 605 kB 00:00 2023-10-14T11:55:47.163 INFO:teuthology.orchestra.run.smithi078.stdout:(119/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 1.1 MB/s | 72 kB 00:00 2023-10-14T11:55:47.188 INFO:teuthology.orchestra.run.smithi078.stdout:(120/131): fuse-2.9.7-15.el8.x86_64.rpm 1.6 MB/s | 83 kB 00:00 2023-10-14T11:55:47.214 INFO:teuthology.orchestra.run.smithi078.stdout:(121/131): elfutils-libelf-devel-0.186-1.el8.x8 1.2 MB/s | 60 kB 00:00 2023-10-14T11:55:47.239 INFO:teuthology.orchestra.run.smithi078.stdout:(122/131): libuuid-devel-2.32.1-35.el8.x86_64.r 1.9 MB/s | 98 kB 00:00 2023-10-14T11:55:47.264 INFO:teuthology.orchestra.run.smithi078.stdout:(123/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 1.1 MB/s | 58 kB 00:00 2023-10-14T11:55:47.340 INFO:teuthology.orchestra.run.smithi078.stdout:(124/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 8.7 MB/s | 2.3 MB 00:00 2023-10-14T11:55:47.365 INFO:teuthology.orchestra.run.smithi078.stdout:(125/131): libcom_err-devel-1.45.6-4.el8.x86_64 307 kB/s | 39 kB 00:00 2023-10-14T11:55:47.390 INFO:teuthology.orchestra.run.smithi078.stdout:(126/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 3.3 MB/s | 171 kB 00:00 2023-10-14T11:55:47.416 INFO:teuthology.orchestra.run.smithi078.stdout:(127/131): NetworkManager-initscripts-updown-1. 2.7 MB/s | 138 kB 00:00 2023-10-14T11:55:47.441 INFO:teuthology.orchestra.run.smithi078.stdout:(128/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 3.6 MB/s | 644 kB 00:00 2023-10-14T11:55:47.508 INFO:teuthology.orchestra.run.smithi078.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 12 MB/s | 798 kB 00:00 2023-10-14T11:55:47.933 INFO:teuthology.orchestra.run.smithi078.stdout:(130/131): librbd1-18.0.0-6730.g4ba8a190.el8.x8 7.2 MB/s | 3.7 MB 00:00 2023-10-14T11:55:48.333 INFO:teuthology.orchestra.run.smithi078.stdout:(131/131): librados2-18.0.0-6730.g4ba8a190.el8. 4.0 MB/s | 3.7 MB 00:00 2023-10-14T11:55:48.336 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2023-10-14T11:55:48.336 INFO:teuthology.orchestra.run.smithi078.stdout:Total 13 MB/s | 209 MB 00:15 2023-10-14T11:55:49.169 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2023-10-14T11:55:49.288 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2023-10-14T11:55:49.288 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2023-10-14T11:55:50.413 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2023-10-14T11:55:50.417 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2023-10-14T11:55:50.782 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2023-10-14T11:55:50.975 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-10-14T11:55:51.151 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-10-14T11:55:51.362 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-10-14T11:55:51.530 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-10-14T11:55:51.671 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-10-14T11:55:51.864 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-10-14T11:55:52.277 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-10-14T11:55:52.488 INFO:teuthology.orchestra.run.smithi195.stdout: Upgrading : librados2-2:18.0.0-6730.g4ba8a190.el8.x86_64 6/134 2023-10-14T11:55:52.888 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librados2-2:18.0.0-6730.g4ba8a190.el8.x86_64 6/134 2023-10-14T11:55:53.088 INFO:teuthology.orchestra.run.smithi195.stdout: Upgrading : librbd1-2:18.0.0-6730.g4ba8a190.el8.x86_64 7/134 2023-10-14T11:55:53.221 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librbd1-2:18.0.0-6730.g4ba8a190.el8.x86_64 7/134 2023-10-14T11:55:53.405 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephfs2-2:18.0.0-6730.g4ba8a190.el8.x86_64 8/134 2023-10-14T11:55:53.504 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libcephfs2-2:18.0.0-6730.g4ba8a190.el8.x86_64 8/134 2023-10-14T11:55:53.730 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rados-2:18.0.0-6730.g4ba8a190.el8.x86_64 9/134 2023-10-14T11:55:53.966 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-10-14T11:55:54.199 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-10-14T11:55:54.584 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-10-14T11:55:54.854 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-10-14T11:55:55.029 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephsqlite-2:18.0.0-6730.g4ba8a190.el8.x86_64 14/134 2023-10-14T11:55:55.138 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6730.g4ba8a190.el8.x86_64 14/134 2023-10-14T11:55:55.421 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libradosstriper1-2:18.0.0-6730.g4ba8a190.el8.x86 15/134 2023-10-14T11:55:55.525 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6730.g4ba8a190.el8.x86 15/134 2023-10-14T11:55:55.545 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2023-10-14T11:55:55.646 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-10-14T11:55:55.791 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-10-14T11:55:55.862 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-10-14T11:55:55.938 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-10-14T11:55:56.020 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-10-14T11:55:56.263 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-10-14T11:55:56.439 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-10-14T11:55:56.557 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-10-14T11:55:56.593 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-10-14T11:55:56.762 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-10-14T11:55:56.797 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-10-14T11:55:57.229 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-10-14T11:55:57.356 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-10-14T11:55:57.447 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : librados2-2:18.0.0-6730.g4ba8a190.el8.x86_64 6/134 2023-10-14T11:55:57.546 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-10-14T11:55:57.857 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librados2-2:18.0.0-6730.g4ba8a190.el8.x86_64 6/134 2023-10-14T11:55:58.080 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : librbd1-2:18.0.0-6730.g4ba8a190.el8.x86_64 7/134 2023-10-14T11:55:58.214 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librbd1-2:18.0.0-6730.g4ba8a190.el8.x86_64 7/134 2023-10-14T11:55:58.254 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-10-14T11:55:58.414 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephfs2-2:18.0.0-6730.g4ba8a190.el8.x86_64 8/134 2023-10-14T11:55:58.510 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-10-14T11:55:58.515 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libcephfs2-2:18.0.0-6730.g4ba8a190.el8.x86_64 8/134 2023-10-14T11:55:58.741 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rados-2:18.0.0-6730.g4ba8a190.el8.x86_64 9/134 2023-10-14T11:55:58.763 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-10-14T11:55:58.957 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-10-14T11:55:59.017 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-10-14T11:55:59.134 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-10-14T11:55:59.233 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-10-14T11:55:59.376 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-10-14T11:55:59.597 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-10-14T11:55:59.675 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-10-14T11:55:59.768 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-ceph-common-2:18.0.0-6730.g4ba8a190.el8. 26/134 2023-10-14T11:55:59.929 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-10-14T11:55:59.964 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-ceph-argparse-2:18.0.0-6730.g4ba8a190.el 27/134 2023-10-14T11:56:00.138 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephsqlite-2:18.0.0-6730.g4ba8a190.el8.x86_64 14/134 2023-10-14T11:56:00.257 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6730.g4ba8a190.el8.x86_64 14/134 2023-10-14T11:56:00.455 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libradosstriper1-2:18.0.0-6730.g4ba8a190.el8.x86 15/134 2023-10-14T11:56:00.543 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6730.g4ba8a190.el8.x86 15/134 2023-10-14T11:56:00.869 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cephfs-2:18.0.0-6730.g4ba8a190.el8.x86_6 28/134 2023-10-14T11:56:00.951 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-10-14T11:56:01.085 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-10-14T11:56:01.175 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-10-14T11:56:01.247 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-10-14T11:56:01.455 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-10-14T11:56:01.690 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-10-14T11:56:01.885 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-10-14T11:56:01.903 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-10-14T11:56:02.080 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-10-14T11:56:02.669 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-10-14T11:56:02.863 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-10-14T11:56:03.558 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-10-14T11:56:03.650 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-10-14T11:56:03.845 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-10-14T11:56:04.008 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-10-14T11:56:04.082 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-10-14T11:56:04.223 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-10-14T11:56:04.300 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-10-14T11:56:04.455 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-10-14T11:56:04.502 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-10-14T11:56:04.634 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rbd-2:18.0.0-6730.g4ba8a190.el8.x86_64 36/134 2023-10-14T11:56:04.702 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-10-14T11:56:04.799 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librados-devel-2:18.0.0-6730.g4ba8a190.el8.x86_6 37/134 2023-10-14T11:56:04.940 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-10-14T11:56:04.971 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-10-14T11:56:05.127 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-ceph-common-2:18.0.0-6730.g4ba8a190.el8. 26/134 2023-10-14T11:56:05.197 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-10-14T11:56:05.265 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-ceph-argparse-2:18.0.0-6730.g4ba8a190.el 27/134 2023-10-14T11:56:05.380 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-10-14T11:56:05.515 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-10-14T11:56:05.752 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-10-14T11:56:05.967 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-10-14T11:56:06.219 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cephfs-2:18.0.0-6730.g4ba8a190.el8.x86_6 28/134 2023-10-14T11:56:06.587 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-10-14T11:56:06.700 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-10-14T11:56:06.890 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-10-14T11:56:07.005 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-10-14T11:56:07.156 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-10-14T11:56:07.195 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-10-14T11:56:07.363 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-10-14T11:56:07.405 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-10-14T11:56:07.598 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-10-14T11:56:07.909 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-10-14T11:56:08.135 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-10-14T11:56:08.379 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-10-14T11:56:08.590 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-10-14T11:56:08.766 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-10-14T11:56:08.972 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-10-14T11:56:09.124 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-10-14T11:56:09.269 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-10-14T11:56:09.488 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-10-14T11:56:09.506 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-10-14T11:56:09.707 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-10-14T11:56:09.748 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-10-14T11:56:09.889 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-10-14T11:56:09.975 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-10-14T11:56:10.085 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rbd-2:18.0.0-6730.g4ba8a190.el8.x86_64 36/134 2023-10-14T11:56:10.174 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-10-14T11:56:10.283 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librados-devel-2:18.0.0-6730.g4ba8a190.el8.x86_6 37/134 2023-10-14T11:56:10.331 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-10-14T11:56:10.456 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-10-14T11:56:10.510 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-10-14T11:56:10.581 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-10-14T11:56:10.678 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-10-14T11:56:10.797 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-10-14T11:56:10.839 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-10-14T11:56:11.007 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-10-14T11:56:11.022 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-10-14T11:56:11.177 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-10-14T11:56:11.201 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-10-14T11:56:11.259 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-10-14T11:56:11.433 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-10-14T11:56:11.506 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-10-14T11:56:11.686 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-10-14T11:56:11.822 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-10-14T11:56:11.851 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-10-14T11:56:12.037 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-10-14T11:56:12.140 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-10-14T11:56:12.326 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-10-14T11:56:12.372 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-10-14T11:56:12.550 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-10-14T11:56:12.590 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-10-14T11:56:12.747 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-10-14T11:56:12.833 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-10-14T11:56:12.904 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-10-14T11:56:13.101 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-10-14T11:56:13.170 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-10-14T11:56:13.295 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-10-14T11:56:13.444 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-10-14T11:56:13.478 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-10-14T11:56:13.621 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-10-14T11:56:13.634 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-10-14T11:56:13.748 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-10-14T11:56:13.875 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-10-14T11:56:13.951 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-10-14T11:56:14.045 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-10-14T11:56:14.179 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-10-14T11:56:14.240 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-10-14T11:56:14.365 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-10-14T11:56:14.594 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-10-14T11:56:14.619 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-10-14T11:56:14.857 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-10-14T11:56:14.866 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-10-14T11:56:15.057 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-10-14T11:56:15.082 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-10-14T11:56:15.270 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-10-14T11:56:15.335 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-10-14T11:56:15.535 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-10-14T11:56:15.688 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-10-14T11:56:15.878 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-10-14T11:56:15.947 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-10-14T11:56:16.205 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-10-14T11:56:16.367 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-10-14T11:56:16.431 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-10-14T11:56:16.620 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-10-14T11:56:16.647 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-10-14T11:56:16.700 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-10-14T11:56:16.926 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-10-14T11:56:16.941 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-10-14T11:56:17.112 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-10-14T11:56:17.234 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-10-14T11:56:17.302 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-10-14T11:56:17.473 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-10-14T11:56:17.512 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-10-14T11:56:17.695 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-10-14T11:56:17.792 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-10-14T11:56:18.035 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-10-14T11:56:18.182 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-10-14T11:56:18.364 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-10-14T11:56:18.541 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-10-14T11:56:18.571 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-10-14T11:56:18.726 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librgw2-2:18.0.0-6730.g4ba8a190.el8.x86_64 92/134 2023-10-14T11:56:18.791 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-10-14T11:56:18.802 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librgw2-2:18.0.0-6730.g4ba8a190.el8.x86_64 92/134 2023-10-14T11:56:18.976 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rgw-2:18.0.0-6730.g4ba8a190.el8.x86_64 93/134 2023-10-14T11:56:19.006 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-10-14T11:56:19.156 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-10-14T11:56:19.326 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-10-14T11:56:19.515 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-10-14T11:56:19.765 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-10-14T11:56:19.958 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-10-14T11:56:20.232 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-10-14T11:56:20.494 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-10-14T11:56:20.643 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-10-14T11:56:20.876 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-10-14T11:56:21.983 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-10-14T11:56:22.272 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-10-14T11:56:22.569 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-10-14T11:56:22.747 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-common-2:18.0.0-6730.g4ba8a190.el8.x86_64 94/134 2023-10-14T11:56:22.843 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-10-14T11:56:22.968 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-common-2:18.0.0-6730.g4ba8a190.el8.x86_64 94/134 2023-10-14T11:56:23.112 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-10-14T11:56:23.333 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-10-14T11:56:23.560 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-common-2:18.0.0-6730.g4ba8a190.el8.x86_64 94/134 2023-10-14T11:56:23.868 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-base-2:18.0.0-6730.g4ba8a190.el8.x86_64 95/134 2023-10-14T11:56:23.938 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-base-2:18.0.0-6730.g4ba8a190.el8.x86_64 95/134 2023-10-14T11:56:24.092 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-selinux-2:18.0.0-6730.g4ba8a190.el8.x86_64 96/134 2023-10-14T11:56:24.168 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-10-14T11:56:24.335 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librgw2-2:18.0.0-6730.g4ba8a190.el8.x86_64 92/134 2023-10-14T11:56:24.414 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librgw2-2:18.0.0-6730.g4ba8a190.el8.x86_64 92/134 2023-10-14T11:56:24.602 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rgw-2:18.0.0-6730.g4ba8a190.el8.x86_64 93/134 2023-10-14T11:56:28.384 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-common-2:18.0.0-6730.g4ba8a190.el8.x86_64 94/134 2023-10-14T11:56:28.602 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-common-2:18.0.0-6730.g4ba8a190.el8.x86_64 94/134 2023-10-14T11:56:29.198 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-common-2:18.0.0-6730.g4ba8a190.el8.x86_64 94/134 2023-10-14T11:56:29.500 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-base-2:18.0.0-6730.g4ba8a190.el8.x86_64 95/134 2023-10-14T11:56:29.566 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-base-2:18.0.0-6730.g4ba8a190.el8.x86_64 95/134 2023-10-14T11:56:29.743 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-selinux-2:18.0.0-6730.g4ba8a190.el8.x86_64 96/134 2023-10-14T11:56:42.929 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6730.g4ba8a190.el8.x86_64 96/134 2023-10-14T11:56:42.929 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /sys 2023-10-14T11:56:42.929 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /proc 2023-10-14T11:56:42.930 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /dev 2023-10-14T11:56:42.930 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /run 2023-10-14T11:56:42.930 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /mnt 2023-10-14T11:56:42.931 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /var/tmp 2023-10-14T11:56:42.931 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /home 2023-10-14T11:56:42.931 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /tmp 2023-10-14T11:56:42.931 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /dev 2023-10-14T11:56:42.932 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-14T11:56:43.289 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-volume-2:18.0.0-6730.g4ba8a190.el8.noarch 97/134 2023-10-14T11:56:44.876 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-volume-2:18.0.0-6730.g4ba8a190.el8.noarch 97/134 2023-10-14T11:56:45.048 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-osd-2:18.0.0-6730.g4ba8a190.el8.x86_64 98/134 2023-10-14T11:56:45.267 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-osd-2:18.0.0-6730.g4ba8a190.el8.x86_64 98/134 2023-10-14T11:56:45.455 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mds-2:18.0.0-6730.g4ba8a190.el8.x86_64 99/134 2023-10-14T11:56:45.923 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mds-2:18.0.0-6730.g4ba8a190.el8.x86_64 99/134 2023-10-14T11:56:46.113 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mon-2:18.0.0-6730.g4ba8a190.el8.x86_64 100/134 2023-10-14T11:56:46.130 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mon-2:18.0.0-6730.g4ba8a190.el8.x86_64 100/134 2023-10-14T11:56:47.840 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: cephadm-2:18.0.0-6730.g4ba8a190.el8.noarch 101/134 2023-10-14T11:56:48.042 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : cephadm-2:18.0.0-6730.g4ba8a190.el8.noarch 101/134 2023-10-14T11:56:48.177 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6730.g4ba8a190.e 102/134 2023-10-14T11:56:48.388 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6730.g4ba8a190. 103/134 2023-10-14T11:56:48.605 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6730.g4ba8a190.el8.noa 104/134 2023-10-14T11:56:48.736 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6730.g4ba8a190.el8.x86_64 96/134 2023-10-14T11:56:48.737 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /sys 2023-10-14T11:56:48.737 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /proc 2023-10-14T11:56:48.737 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /dev 2023-10-14T11:56:48.737 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /run 2023-10-14T11:56:48.737 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /mnt 2023-10-14T11:56:48.738 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /var/tmp 2023-10-14T11:56:48.738 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /home 2023-10-14T11:56:48.738 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /tmp 2023-10-14T11:56:48.738 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /dev 2023-10-14T11:56:48.738 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:56:48.827 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6730.g4ba8a190.el8.noa 104/134 2023-10-14T11:56:49.090 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-volume-2:18.0.0-6730.g4ba8a190.el8.noarch 97/134 2023-10-14T11:56:49.139 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6730.g4ba8a190.el8.n 105/134 2023-10-14T11:56:50.101 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6730.g4ba8a190.el8.n 105/134 2023-10-14T11:56:50.271 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6730.g4ba 106/134 2023-10-14T11:56:50.361 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6730.g4ba 106/134 2023-10-14T11:56:50.677 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-volume-2:18.0.0-6730.g4ba8a190.el8.noarch 97/134 2023-10-14T11:56:50.781 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6730.g4ba8a190.el 107/134 2023-10-14T11:56:50.831 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-osd-2:18.0.0-6730.g4ba8a190.el8.x86_64 98/134 2023-10-14T11:56:50.955 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-2:18.0.0-6730.g4ba8a190.el8.x86_64 108/134 2023-10-14T11:56:50.990 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6730.g4ba8a190.el8.x86_64 108/134 2023-10-14T11:56:51.050 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-osd-2:18.0.0-6730.g4ba8a190.el8.x86_64 98/134 2023-10-14T11:56:51.190 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mds-2:18.0.0-6730.g4ba8a190.el8.x86_64 99/134 2023-10-14T11:56:51.204 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-rook-2:18.0.0-6730.g4ba8a190.el8.noarch 109/134 2023-10-14T11:56:51.215 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6730.g4ba8a190.el8.noarch 109/134 2023-10-14T11:56:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mds-2:18.0.0-6730.g4ba8a190.el8.x86_64 99/134 2023-10-14T11:56:51.824 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mon-2:18.0.0-6730.g4ba8a190.el8.x86_64 100/134 2023-10-14T11:56:51.841 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mon-2:18.0.0-6730.g4ba8a190.el8.x86_64 100/134 2023-10-14T11:56:52.541 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-2:18.0.0-6730.g4ba8a190.el8.x86_64 110/134 2023-10-14T11:56:52.752 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-radosgw-2:18.0.0-6730.g4ba8a190.el8.x86_64 111/134 2023-10-14T11:56:52.785 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6730.g4ba8a190.el8.x86_64 111/134 2023-10-14T11:56:52.938 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6730.g4ba8a 112/134 2023-10-14T11:56:52.980 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6730.g4ba8a 112/134 2023-10-14T11:56:53.147 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : cephfs-mirror-2:18.0.0-6730.g4ba8a190.el8.x86_64 113/134 2023-10-14T11:56:53.517 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-6730.g4ba8a190.el8.x86_64 113/134 2023-10-14T11:56:53.627 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: cephadm-2:18.0.0-6730.g4ba8a190.el8.noarch 101/134 2023-10-14T11:56:53.712 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-mirror-2:18.0.0-6730.g4ba8a190.el8.x86_64 114/134 2023-10-14T11:56:53.833 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephadm-2:18.0.0-6730.g4ba8a190.el8.noarch 101/134 2023-10-14T11:56:53.946 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6730.g4ba8a190.e 102/134 2023-10-14T11:56:54.156 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6730.g4ba8a190. 103/134 2023-10-14T11:56:54.331 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6730.g4ba8a190.el8.noa 104/134 2023-10-14T11:56:54.554 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6730.g4ba8a190.el8.noa 104/134 2023-10-14T11:56:54.855 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6730.g4ba8a190.el8.n 105/134 2023-10-14T11:56:55.814 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6730.g4ba8a190.el8.n 105/134 2023-10-14T11:56:55.973 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6730.g4ba 106/134 2023-10-14T11:56:56.064 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6730.g4ba 106/134 2023-10-14T11:56:56.463 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6730.g4ba8a190.el 107/134 2023-10-14T11:56:56.655 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-2:18.0.0-6730.g4ba8a190.el8.x86_64 108/134 2023-10-14T11:56:56.690 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6730.g4ba8a190.el8.x86_64 108/134 2023-10-14T11:56:56.872 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-rook-2:18.0.0-6730.g4ba8a190.el8.noarch 109/134 2023-10-14T11:56:56.883 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6730.g4ba8a190.el8.noarch 109/134 2023-10-14T11:56:58.220 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-2:18.0.0-6730.g4ba8a190.el8.x86_64 110/134 2023-10-14T11:56:58.347 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6730.g4ba8a190.el8.x86_64 114/134 2023-10-14T11:56:58.440 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-radosgw-2:18.0.0-6730.g4ba8a190.el8.x86_64 111/134 2023-10-14T11:56:58.474 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6730.g4ba8a190.el8.x86_64 111/134 2023-10-14T11:56:58.654 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-test-2:18.0.0-6730.g4ba8a190.el8.x86_64 115/134 2023-10-14T11:56:58.656 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6730.g4ba8a 112/134 2023-10-14T11:56:58.698 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6730.g4ba8a 112/134 2023-10-14T11:56:58.819 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-10-14T11:56:58.873 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephfs-mirror-2:18.0.0-6730.g4ba8a190.el8.x86_64 113/134 2023-10-14T11:56:59.007 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-10-14T11:56:59.246 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-6730.g4ba8a190.el8.x86_64 113/134 2023-10-14T11:56:59.406 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-mirror-2:18.0.0-6730.g4ba8a190.el8.x86_64 114/134 2023-10-14T11:56:59.873 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-10-14T11:57:00.231 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-10-14T11:57:00.409 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-fuse-2:18.0.0-6730.g4ba8a190.el8.x86_64 119/134 2023-10-14T11:57:00.582 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-10-14T11:57:00.782 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephfs-devel-2:18.0.0-6730.g4ba8a190.el8.x86_ 121/134 2023-10-14T11:57:00.929 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-10-14T11:57:03.128 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-14T11:57:03.386 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-14T11:57:03.403 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-14T11:57:03.553 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : cephfs-top-2:18.0.0-6730.g4ba8a190.el8.noarch 124/134 2023-10-14T11:57:03.844 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-fuse-2:18.0.0-6730.g4ba8a190.el8.x86_64 125/134 2023-10-14T11:57:04.059 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6730.g4ba8a190.el8.x86_64 114/134 2023-10-14T11:57:04.174 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-nbd-2:18.0.0-6730.g4ba8a190.el8.x86_64 126/134 2023-10-14T11:57:04.375 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-test-2:18.0.0-6730.g4ba8a190.el8.x86_64 115/134 2023-10-14T11:57:04.479 INFO:teuthology.orchestra.run.smithi195.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-10-14T11:57:04.530 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-10-14T11:57:04.661 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-10-14T11:57:04.669 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-10-14T11:57:04.728 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-10-14T11:57:04.854 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-10-14T11:57:05.052 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-10-14T11:57:05.148 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-10-14T11:57:05.378 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-10-14T11:57:05.390 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-10-14T11:57:05.391 INFO:teuthology.orchestra.run.smithi195.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-14T11:57:05.588 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-14T11:57:05.589 INFO:teuthology.orchestra.run.smithi195.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-10-14T11:57:05.654 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-10-14T11:57:05.816 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-10-14T11:57:06.017 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-10-14T11:57:06.017 INFO:teuthology.orchestra.run.smithi195.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-10-14T11:57:06.180 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-fuse-2:18.0.0-6730.g4ba8a190.el8.x86_64 119/134 2023-10-14T11:57:06.401 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-10-14T11:57:06.517 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-10-14T11:57:06.517 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-2:18.0.0-6730.g4ba8a190.el8.x86_64 1/134 2023-10-14T11:57:06.517 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-base-2:18.0.0-6730.g4ba8a190.el8.x86_64 2/134 2023-10-14T11:57:06.518 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-common-2:18.0.0-6730.g4ba8a190.el8.x86_64 3/134 2023-10-14T11:57:06.518 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-fuse-2:18.0.0-6730.g4ba8a190.el8.x86_64 4/134 2023-10-14T11:57:06.518 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6730.g4ba8a 5/134 2023-10-14T11:57:06.518 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mds-2:18.0.0-6730.g4ba8a190.el8.x86_64 6/134 2023-10-14T11:57:06.518 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-2:18.0.0-6730.g4ba8a190.el8.x86_64 7/134 2023-10-14T11:57:06.519 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mon-2:18.0.0-6730.g4ba8a190.el8.x86_64 8/134 2023-10-14T11:57:06.519 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-osd-2:18.0.0-6730.g4ba8a190.el8.x86_64 9/134 2023-10-14T11:57:06.519 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-radosgw-2:18.0.0-6730.g4ba8a190.el8.x86_64 10/134 2023-10-14T11:57:06.519 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-selinux-2:18.0.0-6730.g4ba8a190.el8.x86_64 11/134 2023-10-14T11:57:06.520 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-test-2:18.0.0-6730.g4ba8a190.el8.x86_64 12/134 2023-10-14T11:57:06.520 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : cephfs-mirror-2:18.0.0-6730.g4ba8a190.el8.x86_64 13/134 2023-10-14T11:57:06.520 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephfs-devel-2:18.0.0-6730.g4ba8a190.el8.x86_ 14/134 2023-10-14T11:57:06.520 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephfs2-2:18.0.0-6730.g4ba8a190.el8.x86_64 15/134 2023-10-14T11:57:06.520 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephsqlite-2:18.0.0-6730.g4ba8a190.el8.x86_64 16/134 2023-10-14T11:57:06.520 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados-devel-2:18.0.0-6730.g4ba8a190.el8.x86_6 17/134 2023-10-14T11:57:06.521 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libradosstriper1-2:18.0.0-6730.g4ba8a190.el8.x86 18/134 2023-10-14T11:57:06.521 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librgw2-2:18.0.0-6730.g4ba8a190.el8.x86_64 19/134 2023-10-14T11:57:06.521 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6730.g4ba8a190.el 20/134 2023-10-14T11:57:06.521 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-ceph-common-2:18.0.0-6730.g4ba8a190.el8. 21/134 2023-10-14T11:57:06.521 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cephfs-2:18.0.0-6730.g4ba8a190.el8.x86_6 22/134 2023-10-14T11:57:06.522 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rados-2:18.0.0-6730.g4ba8a190.el8.x86_64 23/134 2023-10-14T11:57:06.522 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rbd-2:18.0.0-6730.g4ba8a190.el8.x86_64 24/134 2023-10-14T11:57:06.522 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rgw-2:18.0.0-6730.g4ba8a190.el8.x86_64 25/134 2023-10-14T11:57:06.522 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-fuse-2:18.0.0-6730.g4ba8a190.el8.x86_64 26/134 2023-10-14T11:57:06.522 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-mirror-2:18.0.0-6730.g4ba8a190.el8.x86_64 27/134 2023-10-14T11:57:06.523 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-nbd-2:18.0.0-6730.g4ba8a190.el8.x86_64 28/134 2023-10-14T11:57:06.523 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6730.g4ba8a190. 29/134 2023-10-14T11:57:06.537 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6730.g4ba8a190.el8.noa 30/134 2023-10-14T11:57:06.538 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6730.g4ba8a190.el8.n 31/134 2023-10-14T11:57:06.538 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6730.g4ba 32/134 2023-10-14T11:57:06.538 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6730.g4ba8a190.el 33/134 2023-10-14T11:57:06.539 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6730.g4ba8a190.el8.noarch 34/134 2023-10-14T11:57:06.539 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6730.g4ba8a190.e 35/134 2023-10-14T11:57:06.539 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-volume-2:18.0.0-6730.g4ba8a190.el8.noarch 36/134 2023-10-14T11:57:06.539 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : cephadm-2:18.0.0-6730.g4ba8a190.el8.noarch 37/134 2023-10-14T11:57:06.539 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : cephfs-top-2:18.0.0-6730.g4ba8a190.el8.noarch 38/134 2023-10-14T11:57:06.540 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-10-14T11:57:06.540 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-10-14T11:57:06.540 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-10-14T11:57:06.540 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-10-14T11:57:06.540 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-10-14T11:57:06.540 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-10-14T11:57:06.541 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-10-14T11:57:06.541 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-10-14T11:57:06.541 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-10-14T11:57:06.541 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-10-14T11:57:06.542 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-10-14T11:57:06.542 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-10-14T11:57:06.542 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-10-14T11:57:06.542 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-10-14T11:57:06.542 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-10-14T11:57:06.543 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-10-14T11:57:06.543 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-10-14T11:57:06.543 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-10-14T11:57:06.543 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-10-14T11:57:06.543 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-10-14T11:57:06.543 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-10-14T11:57:06.544 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-10-14T11:57:06.544 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-10-14T11:57:06.544 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-10-14T11:57:06.544 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-10-14T11:57:06.544 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-10-14T11:57:06.545 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-10-14T11:57:06.545 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-10-14T11:57:06.545 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-10-14T11:57:06.545 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-10-14T11:57:06.545 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-10-14T11:57:06.546 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-10-14T11:57:06.546 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-10-14T11:57:06.546 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-10-14T11:57:06.546 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-10-14T11:57:06.546 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-10-14T11:57:06.547 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-10-14T11:57:06.547 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-10-14T11:57:06.547 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-10-14T11:57:06.547 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-10-14T11:57:06.547 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-10-14T11:57:06.548 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-10-14T11:57:06.548 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-10-14T11:57:06.548 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-10-14T11:57:06.548 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-10-14T11:57:06.548 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-10-14T11:57:06.549 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-10-14T11:57:06.549 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-10-14T11:57:06.549 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-10-14T11:57:06.549 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-10-14T11:57:06.549 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-10-14T11:57:06.550 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-10-14T11:57:06.550 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-10-14T11:57:06.550 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-10-14T11:57:06.550 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-10-14T11:57:06.550 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-10-14T11:57:06.551 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-10-14T11:57:06.551 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-10-14T11:57:06.551 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-10-14T11:57:06.551 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-10-14T11:57:06.551 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-10-14T11:57:06.551 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-10-14T11:57:06.552 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-10-14T11:57:06.552 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-10-14T11:57:06.552 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-10-14T11:57:06.552 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-10-14T11:57:06.552 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-10-14T11:57:06.553 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-10-14T11:57:06.553 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-10-14T11:57:06.553 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-10-14T11:57:06.553 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-10-14T11:57:06.553 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-10-14T11:57:06.554 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-10-14T11:57:06.554 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-10-14T11:57:06.554 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-10-14T11:57:06.554 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-10-14T11:57:06.554 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-10-14T11:57:06.555 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-10-14T11:57:06.555 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-10-14T11:57:06.555 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-10-14T11:57:06.555 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-10-14T11:57:06.555 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-10-14T11:57:06.556 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-10-14T11:57:06.556 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-10-14T11:57:06.556 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-10-14T11:57:06.556 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-10-14T11:57:06.556 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-10-14T11:57:06.556 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-10-14T11:57:06.557 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-10-14T11:57:06.557 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-10-14T11:57:06.557 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados2-2:18.0.0-6730.g4ba8a190.el8.x86_64 129/134 2023-10-14T11:57:06.557 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-10-14T11:57:06.557 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librbd1-2:18.0.0-6730.g4ba8a190.el8.x86_64 131/134 2023-10-14T11:57:06.558 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-14T11:57:06.558 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-10-14T11:57:06.575 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephfs-devel-2:18.0.0-6730.g4ba8a190.el8.x86_ 121/134 2023-10-14T11:57:06.756 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-10-14T11:57:08.905 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-14T11:57:09.204 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-14T11:57:09.223 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-14T11:57:09.284 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-10-14T11:57:09.284 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:49 ago on Sat 14 Oct 2023 11:55:20 AM UTC. 2023-10-14T11:57:09.380 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephfs-top-2:18.0.0-6730.g4ba8a190.el8.noarch 124/134 2023-10-14T11:57:09.688 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-fuse-2:18.0.0-6730.g4ba8a190.el8.x86_64 125/134 2023-10-14T11:57:10.011 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-nbd-2:18.0.0-6730.g4ba8a190.el8.x86_64 126/134 2023-10-14T11:57:10.340 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-10-14T11:57:10.471 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-10-14T11:57:10.480 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-10-14T11:57:10.679 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-10-14T11:57:10.896 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-10-14T11:57:10.974 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-10-14T11:57:11.229 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-10-14T11:57:11.242 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-10-14T11:57:11.243 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-14T11:57:11.432 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-14T11:57:11.432 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-10-14T11:57:11.699 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-10-14T11:57:11.920 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-10-14T11:57:12.400 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-10-14T11:57:12.401 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-2:18.0.0-6730.g4ba8a190.el8.x86_64 1/134 2023-10-14T11:57:12.401 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-base-2:18.0.0-6730.g4ba8a190.el8.x86_64 2/134 2023-10-14T11:57:12.401 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-common-2:18.0.0-6730.g4ba8a190.el8.x86_64 3/134 2023-10-14T11:57:12.401 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-fuse-2:18.0.0-6730.g4ba8a190.el8.x86_64 4/134 2023-10-14T11:57:12.402 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6730.g4ba8a 5/134 2023-10-14T11:57:12.402 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mds-2:18.0.0-6730.g4ba8a190.el8.x86_64 6/134 2023-10-14T11:57:12.402 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-2:18.0.0-6730.g4ba8a190.el8.x86_64 7/134 2023-10-14T11:57:12.402 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mon-2:18.0.0-6730.g4ba8a190.el8.x86_64 8/134 2023-10-14T11:57:12.402 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-osd-2:18.0.0-6730.g4ba8a190.el8.x86_64 9/134 2023-10-14T11:57:12.403 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-radosgw-2:18.0.0-6730.g4ba8a190.el8.x86_64 10/134 2023-10-14T11:57:12.403 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-selinux-2:18.0.0-6730.g4ba8a190.el8.x86_64 11/134 2023-10-14T11:57:12.403 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-test-2:18.0.0-6730.g4ba8a190.el8.x86_64 12/134 2023-10-14T11:57:12.404 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephfs-mirror-2:18.0.0-6730.g4ba8a190.el8.x86_64 13/134 2023-10-14T11:57:12.404 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephfs-devel-2:18.0.0-6730.g4ba8a190.el8.x86_ 14/134 2023-10-14T11:57:12.404 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephfs2-2:18.0.0-6730.g4ba8a190.el8.x86_64 15/134 2023-10-14T11:57:12.404 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephsqlite-2:18.0.0-6730.g4ba8a190.el8.x86_64 16/134 2023-10-14T11:57:12.404 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados-devel-2:18.0.0-6730.g4ba8a190.el8.x86_6 17/134 2023-10-14T11:57:12.405 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libradosstriper1-2:18.0.0-6730.g4ba8a190.el8.x86 18/134 2023-10-14T11:57:12.405 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librgw2-2:18.0.0-6730.g4ba8a190.el8.x86_64 19/134 2023-10-14T11:57:12.405 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6730.g4ba8a190.el 20/134 2023-10-14T11:57:12.405 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-ceph-common-2:18.0.0-6730.g4ba8a190.el8. 21/134 2023-10-14T11:57:12.405 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cephfs-2:18.0.0-6730.g4ba8a190.el8.x86_6 22/134 2023-10-14T11:57:12.406 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rados-2:18.0.0-6730.g4ba8a190.el8.x86_64 23/134 2023-10-14T11:57:12.406 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rbd-2:18.0.0-6730.g4ba8a190.el8.x86_64 24/134 2023-10-14T11:57:12.406 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rgw-2:18.0.0-6730.g4ba8a190.el8.x86_64 25/134 2023-10-14T11:57:12.406 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-fuse-2:18.0.0-6730.g4ba8a190.el8.x86_64 26/134 2023-10-14T11:57:12.406 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-mirror-2:18.0.0-6730.g4ba8a190.el8.x86_64 27/134 2023-10-14T11:57:12.407 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-nbd-2:18.0.0-6730.g4ba8a190.el8.x86_64 28/134 2023-10-14T11:57:12.407 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6730.g4ba8a190. 29/134 2023-10-14T11:57:12.407 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6730.g4ba8a190.el8.noa 30/134 2023-10-14T11:57:12.417 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6730.g4ba8a190.el8.n 31/134 2023-10-14T11:57:12.417 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6730.g4ba 32/134 2023-10-14T11:57:12.417 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6730.g4ba8a190.el 33/134 2023-10-14T11:57:12.417 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6730.g4ba8a190.el8.noarch 34/134 2023-10-14T11:57:12.418 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6730.g4ba8a190.e 35/134 2023-10-14T11:57:12.418 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-volume-2:18.0.0-6730.g4ba8a190.el8.noarch 36/134 2023-10-14T11:57:12.418 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephadm-2:18.0.0-6730.g4ba8a190.el8.noarch 37/134 2023-10-14T11:57:12.418 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephfs-top-2:18.0.0-6730.g4ba8a190.el8.noarch 38/134 2023-10-14T11:57:12.418 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-10-14T11:57:12.419 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-10-14T11:57:12.419 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-10-14T11:57:12.419 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-10-14T11:57:12.419 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-10-14T11:57:12.419 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-10-14T11:57:12.420 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-10-14T11:57:12.420 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-10-14T11:57:12.420 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-10-14T11:57:12.420 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-10-14T11:57:12.420 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-10-14T11:57:12.421 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-10-14T11:57:12.421 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-10-14T11:57:12.421 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-10-14T11:57:12.421 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-10-14T11:57:12.421 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-10-14T11:57:12.422 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-10-14T11:57:12.422 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-10-14T11:57:12.422 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-10-14T11:57:12.422 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-10-14T11:57:12.422 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-10-14T11:57:12.423 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-10-14T11:57:12.423 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-10-14T11:57:12.423 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-10-14T11:57:12.424 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-10-14T11:57:12.424 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-10-14T11:57:12.424 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-10-14T11:57:12.424 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-10-14T11:57:12.425 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-10-14T11:57:12.425 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-10-14T11:57:12.425 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-10-14T11:57:12.425 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-10-14T11:57:12.425 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-10-14T11:57:12.426 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-10-14T11:57:12.426 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-10-14T11:57:12.426 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-10-14T11:57:12.426 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-10-14T11:57:12.427 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-10-14T11:57:12.427 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-10-14T11:57:12.427 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-10-14T11:57:12.427 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-10-14T11:57:12.427 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-10-14T11:57:12.428 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-10-14T11:57:12.428 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-10-14T11:57:12.428 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-10-14T11:57:12.428 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-10-14T11:57:12.428 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-10-14T11:57:12.429 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-10-14T11:57:12.429 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-10-14T11:57:12.429 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-10-14T11:57:12.429 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-10-14T11:57:12.430 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-10-14T11:57:12.430 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-10-14T11:57:12.430 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-10-14T11:57:12.430 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-10-14T11:57:12.431 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-10-14T11:57:12.431 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-10-14T11:57:12.431 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-10-14T11:57:12.431 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-10-14T11:57:12.431 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-10-14T11:57:12.432 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-10-14T11:57:12.432 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-10-14T11:57:12.432 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-10-14T11:57:12.432 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-10-14T11:57:12.432 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-10-14T11:57:12.433 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-10-14T11:57:12.433 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-10-14T11:57:12.433 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-10-14T11:57:12.433 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-10-14T11:57:12.433 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-10-14T11:57:12.434 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-10-14T11:57:12.434 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-10-14T11:57:12.434 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-10-14T11:57:12.435 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-10-14T11:57:12.435 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-10-14T11:57:12.435 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-10-14T11:57:12.435 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-10-14T11:57:12.435 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-10-14T11:57:12.436 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-10-14T11:57:12.436 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-10-14T11:57:12.436 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-10-14T11:57:12.436 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-10-14T11:57:12.436 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-10-14T11:57:12.437 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-10-14T11:57:12.437 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-10-14T11:57:12.437 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-10-14T11:57:12.437 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-10-14T11:57:12.437 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-10-14T11:57:12.438 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-10-14T11:57:12.438 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-10-14T11:57:12.438 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados2-2:18.0.0-6730.g4ba8a190.el8.x86_64 129/134 2023-10-14T11:57:12.438 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-10-14T11:57:12.438 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librbd1-2:18.0.0-6730.g4ba8a190.el8.x86_64 131/134 2023-10-14T11:57:12.439 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-14T11:57:12.439 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-10-14T11:57:13.130 INFO:teuthology.orchestra.run.smithi195.stdout:Installed products updated. 2023-10-14T11:57:13.352 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-14T11:57:13.353 INFO:teuthology.orchestra.run.smithi195.stdout:Upgraded: 2023-10-14T11:57:13.353 INFO:teuthology.orchestra.run.smithi195.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-10-14T11:57:13.353 INFO:teuthology.orchestra.run.smithi195.stdout: librados2-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:13.353 INFO:teuthology.orchestra.run.smithi195.stdout: librbd1-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:13.354 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2023-10-14T11:57:13.354 INFO:teuthology.orchestra.run.smithi195.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-10-14T11:57:13.354 INFO:teuthology.orchestra.run.smithi195.stdout: automake-1.16.1-7.el8.noarch 2023-10-14T11:57:13.354 INFO:teuthology.orchestra.run.smithi195.stdout: bison-3.0.4-10.el8.x86_64 2023-10-14T11:57:13.354 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:13.355 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-base-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:13.355 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-common-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:13.355 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-fuse-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:13.355 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-grafana-dashboards-2:18.0.0-6730.g4ba8a190.el8.noarch 2023-10-14T11:57:13.355 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-immutable-object-cache-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:13.356 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mds-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:13.356 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:13.356 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-cephadm-2:18.0.0-6730.g4ba8a190.el8.noarch 2023-10-14T11:57:13.356 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-dashboard-2:18.0.0-6730.g4ba8a190.el8.noarch 2023-10-14T11:57:13.357 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6730.g4ba8a190.el8.noarch 2023-10-14T11:57:13.357 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-modules-core-2:18.0.0-6730.g4ba8a190.el8.noarch 2023-10-14T11:57:13.357 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-rook-2:18.0.0-6730.g4ba8a190.el8.noarch 2023-10-14T11:57:13.357 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mon-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:13.357 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-osd-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:13.358 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-prometheus-alerts-2:18.0.0-6730.g4ba8a190.el8.noarch 2023-10-14T11:57:13.358 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-radosgw-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:13.358 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-selinux-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:13.358 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-test-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:13.358 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-volume-2:18.0.0-6730.g4ba8a190.el8.noarch 2023-10-14T11:57:13.359 INFO:teuthology.orchestra.run.smithi195.stdout: cephadm-2:18.0.0-6730.g4ba8a190.el8.noarch 2023-10-14T11:57:13.359 INFO:teuthology.orchestra.run.smithi195.stdout: cephfs-mirror-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:13.359 INFO:teuthology.orchestra.run.smithi195.stdout: cephfs-top-2:18.0.0-6730.g4ba8a190.el8.noarch 2023-10-14T11:57:13.361 INFO:teuthology.orchestra.run.smithi195.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-10-14T11:57:13.362 INFO:teuthology.orchestra.run.smithi195.stdout: flex-2.6.1-9.el8.x86_64 2023-10-14T11:57:13.362 INFO:teuthology.orchestra.run.smithi195.stdout: fuse-2.9.7-15.el8.x86_64 2023-10-14T11:57:13.362 INFO:teuthology.orchestra.run.smithi195.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-14T11:57:13.362 INFO:teuthology.orchestra.run.smithi195.stdout: jq-1.6-3.el8.x86_64 2023-10-14T11:57:13.363 INFO:teuthology.orchestra.run.smithi195.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-10-14T11:57:13.363 INFO:teuthology.orchestra.run.smithi195.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-10-14T11:57:13.363 INFO:teuthology.orchestra.run.smithi195.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-10-14T11:57:13.363 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs-devel-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:13.364 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs2-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:13.364 INFO:teuthology.orchestra.run.smithi195.stdout: libcephsqlite-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:13.364 INFO:teuthology.orchestra.run.smithi195.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-10-14T11:57:13.364 INFO:teuthology.orchestra.run.smithi195.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-10-14T11:57:13.364 INFO:teuthology.orchestra.run.smithi195.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-14T11:57:13.365 INFO:teuthology.orchestra.run.smithi195.stdout: libpq-13.5-1.el8.x86_64 2023-10-14T11:57:13.365 INFO:teuthology.orchestra.run.smithi195.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-10-14T11:57:13.365 INFO:teuthology.orchestra.run.smithi195.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-10-14T11:57:13.365 INFO:teuthology.orchestra.run.smithi195.stdout: librados-devel-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:13.365 INFO:teuthology.orchestra.run.smithi195.stdout: libradosstriper1-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:13.366 INFO:teuthology.orchestra.run.smithi195.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-10-14T11:57:13.366 INFO:teuthology.orchestra.run.smithi195.stdout: librgw2-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:13.366 INFO:teuthology.orchestra.run.smithi195.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-10-14T11:57:13.366 INFO:teuthology.orchestra.run.smithi195.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-10-14T11:57:13.367 INFO:teuthology.orchestra.run.smithi195.stdout: libtool-2.4.6-25.el8.x86_64 2023-10-14T11:57:13.367 INFO:teuthology.orchestra.run.smithi195.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-14T11:57:13.367 INFO:teuthology.orchestra.run.smithi195.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-10-14T11:57:13.368 INFO:teuthology.orchestra.run.smithi195.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-10-14T11:57:13.368 INFO:teuthology.orchestra.run.smithi195.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-14T11:57:13.368 INFO:teuthology.orchestra.run.smithi195.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-14T11:57:13.368 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-0.3.15-3.el8.x86_64 2023-10-14T11:57:13.368 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-10-14T11:57:13.369 INFO:teuthology.orchestra.run.smithi195.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-10-14T11:57:13.369 INFO:teuthology.orchestra.run.smithi195.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-10-14T11:57:13.369 INFO:teuthology.orchestra.run.smithi195.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-10-14T11:57:13.369 INFO:teuthology.orchestra.run.smithi195.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-10-14T11:57:13.370 INFO:teuthology.orchestra.run.smithi195.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-14T11:57:13.370 INFO:teuthology.orchestra.run.smithi195.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-14T11:57:13.370 INFO:teuthology.orchestra.run.smithi195.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-14T11:57:13.370 INFO:teuthology.orchestra.run.smithi195.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-10-14T11:57:13.371 INFO:teuthology.orchestra.run.smithi195.stdout: python3-babel-2.5.1-7.el8.noarch 2023-10-14T11:57:13.371 INFO:teuthology.orchestra.run.smithi195.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-14T11:57:13.371 INFO:teuthology.orchestra.run.smithi195.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-14T11:57:13.371 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-14T11:57:13.371 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-argparse-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:13.372 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-common-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:13.372 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cephfs-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:13.372 INFO:teuthology.orchestra.run.smithi195.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-14T11:57:13.372 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-10-14T11:57:13.372 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-14T11:57:13.373 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-14T11:57:13.373 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-10-14T11:57:13.373 INFO:teuthology.orchestra.run.smithi195.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-14T11:57:13.373 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-14T11:57:13.374 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-14T11:57:13.374 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-10-14T11:57:13.374 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-14T11:57:13.375 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-14T11:57:13.375 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-14T11:57:13.375 INFO:teuthology.orchestra.run.smithi195.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-14T11:57:13.375 INFO:teuthology.orchestra.run.smithi195.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-14T11:57:13.376 INFO:teuthology.orchestra.run.smithi195.stdout: python3-mako-1.0.6-13.el8.noarch 2023-10-14T11:57:13.376 INFO:teuthology.orchestra.run.smithi195.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-14T11:57:13.376 INFO:teuthology.orchestra.run.smithi195.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-14T11:57:13.376 INFO:teuthology.orchestra.run.smithi195.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-10-14T11:57:13.376 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-10-14T11:57:13.377 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-10-14T11:57:13.377 INFO:teuthology.orchestra.run.smithi195.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-14T11:57:13.377 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-14T11:57:13.377 INFO:teuthology.orchestra.run.smithi195.stdout: python3-portend-2.6-1.el8.noarch 2023-10-14T11:57:13.377 INFO:teuthology.orchestra.run.smithi195.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-14T11:57:13.378 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-10-14T11:57:13.378 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-14T11:57:13.378 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-14T11:57:13.378 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pycparser-2.14-14.el8.noarch 2023-10-14T11:57:13.378 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pytz-2017.2-9.el8.noarch 2023-10-14T11:57:13.379 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rados-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:13.379 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rbd-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:13.379 INFO:teuthology.orchestra.run.smithi195.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-14T11:57:13.379 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-14T11:57:13.379 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rgw-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:13.380 INFO:teuthology.orchestra.run.smithi195.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-14T11:57:13.380 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-14T11:57:13.380 INFO:teuthology.orchestra.run.smithi195.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-10-14T11:57:13.380 INFO:teuthology.orchestra.run.smithi195.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-14T11:57:13.380 INFO:teuthology.orchestra.run.smithi195.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-14T11:57:13.381 INFO:teuthology.orchestra.run.smithi195.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-14T11:57:13.381 INFO:teuthology.orchestra.run.smithi195.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-14T11:57:13.381 INFO:teuthology.orchestra.run.smithi195.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-14T11:57:13.381 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-14T11:57:13.381 INFO:teuthology.orchestra.run.smithi195.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-14T11:57:13.382 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-14T11:57:13.382 INFO:teuthology.orchestra.run.smithi195.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-14T11:57:13.382 INFO:teuthology.orchestra.run.smithi195.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-14T11:57:13.382 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-fuse-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:13.382 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-mirror-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:13.383 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-nbd-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:13.383 INFO:teuthology.orchestra.run.smithi195.stdout: socat-1.7.4.1-1.el8.x86_64 2023-10-14T11:57:13.383 INFO:teuthology.orchestra.run.smithi195.stdout: thrift-0.13.0-2.el8.x86_64 2023-10-14T11:57:13.383 INFO:teuthology.orchestra.run.smithi195.stdout: uuid-1.6.2-43.el8.x86_64 2023-10-14T11:57:13.383 INFO:teuthology.orchestra.run.smithi195.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-10-14T11:57:13.384 INFO:teuthology.orchestra.run.smithi195.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-14T11:57:13.384 INFO:teuthology.orchestra.run.smithi195.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-10-14T11:57:13.385 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-14T11:57:13.385 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2023-10-14T11:57:13.560 DEBUG:teuthology.parallel:result is None 2023-10-14T11:57:15.267 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-10-14T11:57:15.267 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:54 ago on Sat 14 Oct 2023 11:55:21 AM UTC. 2023-10-14T11:57:20.989 INFO:teuthology.orchestra.run.smithi078.stdout:Installed products updated. 2023-10-14T11:57:21.215 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:57:21.215 INFO:teuthology.orchestra.run.smithi078.stdout:Upgraded: 2023-10-14T11:57:21.216 INFO:teuthology.orchestra.run.smithi078.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-10-14T11:57:21.216 INFO:teuthology.orchestra.run.smithi078.stdout: librados2-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:21.216 INFO:teuthology.orchestra.run.smithi078.stdout: librbd1-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:21.216 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2023-10-14T11:57:21.216 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-10-14T11:57:21.217 INFO:teuthology.orchestra.run.smithi078.stdout: automake-1.16.1-7.el8.noarch 2023-10-14T11:57:21.217 INFO:teuthology.orchestra.run.smithi078.stdout: bison-3.0.4-10.el8.x86_64 2023-10-14T11:57:21.217 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:21.217 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-base-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:21.217 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-common-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:21.218 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-fuse-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:21.218 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-grafana-dashboards-2:18.0.0-6730.g4ba8a190.el8.noarch 2023-10-14T11:57:21.218 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-immutable-object-cache-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:21.218 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mds-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:21.218 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:21.219 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-cephadm-2:18.0.0-6730.g4ba8a190.el8.noarch 2023-10-14T11:57:21.219 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-dashboard-2:18.0.0-6730.g4ba8a190.el8.noarch 2023-10-14T11:57:21.219 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6730.g4ba8a190.el8.noarch 2023-10-14T11:57:21.219 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-modules-core-2:18.0.0-6730.g4ba8a190.el8.noarch 2023-10-14T11:57:21.219 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-rook-2:18.0.0-6730.g4ba8a190.el8.noarch 2023-10-14T11:57:21.220 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mon-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:21.220 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-osd-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:21.220 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-prometheus-alerts-2:18.0.0-6730.g4ba8a190.el8.noarch 2023-10-14T11:57:21.220 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-radosgw-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:21.220 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-selinux-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:21.221 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-test-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:21.221 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-volume-2:18.0.0-6730.g4ba8a190.el8.noarch 2023-10-14T11:57:21.221 INFO:teuthology.orchestra.run.smithi078.stdout: cephadm-2:18.0.0-6730.g4ba8a190.el8.noarch 2023-10-14T11:57:21.221 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-mirror-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:21.221 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-top-2:18.0.0-6730.g4ba8a190.el8.noarch 2023-10-14T11:57:21.222 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-10-14T11:57:21.222 INFO:teuthology.orchestra.run.smithi078.stdout: flex-2.6.1-9.el8.x86_64 2023-10-14T11:57:21.222 INFO:teuthology.orchestra.run.smithi078.stdout: fuse-2.9.7-15.el8.x86_64 2023-10-14T11:57:21.222 INFO:teuthology.orchestra.run.smithi078.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-14T11:57:21.223 INFO:teuthology.orchestra.run.smithi078.stdout: jq-1.6-3.el8.x86_64 2023-10-14T11:57:21.223 INFO:teuthology.orchestra.run.smithi078.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-10-14T11:57:21.223 INFO:teuthology.orchestra.run.smithi078.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-10-14T11:57:21.223 INFO:teuthology.orchestra.run.smithi078.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-10-14T11:57:21.224 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs-devel-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:21.224 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs2-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:21.224 INFO:teuthology.orchestra.run.smithi078.stdout: libcephsqlite-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:21.224 INFO:teuthology.orchestra.run.smithi078.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-10-14T11:57:21.225 INFO:teuthology.orchestra.run.smithi078.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-10-14T11:57:21.225 INFO:teuthology.orchestra.run.smithi078.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-14T11:57:21.225 INFO:teuthology.orchestra.run.smithi078.stdout: libpq-13.5-1.el8.x86_64 2023-10-14T11:57:21.225 INFO:teuthology.orchestra.run.smithi078.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-10-14T11:57:21.225 INFO:teuthology.orchestra.run.smithi078.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-10-14T11:57:21.226 INFO:teuthology.orchestra.run.smithi078.stdout: librados-devel-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:21.226 INFO:teuthology.orchestra.run.smithi078.stdout: libradosstriper1-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:21.226 INFO:teuthology.orchestra.run.smithi078.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-10-14T11:57:21.226 INFO:teuthology.orchestra.run.smithi078.stdout: librgw2-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:21.227 INFO:teuthology.orchestra.run.smithi078.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-10-14T11:57:21.227 INFO:teuthology.orchestra.run.smithi078.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-10-14T11:57:21.227 INFO:teuthology.orchestra.run.smithi078.stdout: libtool-2.4.6-25.el8.x86_64 2023-10-14T11:57:21.227 INFO:teuthology.orchestra.run.smithi078.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-14T11:57:21.228 INFO:teuthology.orchestra.run.smithi078.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-10-14T11:57:21.228 INFO:teuthology.orchestra.run.smithi078.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-10-14T11:57:21.228 INFO:teuthology.orchestra.run.smithi078.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-14T11:57:21.228 INFO:teuthology.orchestra.run.smithi078.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-14T11:57:21.228 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-0.3.15-3.el8.x86_64 2023-10-14T11:57:21.229 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-10-14T11:57:21.229 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-10-14T11:57:21.229 INFO:teuthology.orchestra.run.smithi078.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-10-14T11:57:21.229 INFO:teuthology.orchestra.run.smithi078.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-10-14T11:57:21.230 INFO:teuthology.orchestra.run.smithi078.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-10-14T11:57:21.230 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-14T11:57:21.230 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-14T11:57:21.230 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-14T11:57:21.230 INFO:teuthology.orchestra.run.smithi078.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-10-14T11:57:21.231 INFO:teuthology.orchestra.run.smithi078.stdout: python3-babel-2.5.1-7.el8.noarch 2023-10-14T11:57:21.231 INFO:teuthology.orchestra.run.smithi078.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-14T11:57:21.231 INFO:teuthology.orchestra.run.smithi078.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-14T11:57:21.231 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-14T11:57:21.232 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-argparse-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:21.232 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-common-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:21.232 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cephfs-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:21.232 INFO:teuthology.orchestra.run.smithi078.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-14T11:57:21.233 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-10-14T11:57:21.233 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-14T11:57:21.233 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-14T11:57:21.233 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-10-14T11:57:21.234 INFO:teuthology.orchestra.run.smithi078.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-14T11:57:21.234 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-14T11:57:21.234 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-14T11:57:21.234 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-10-14T11:57:21.235 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-14T11:57:21.235 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-14T11:57:21.235 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-14T11:57:21.235 INFO:teuthology.orchestra.run.smithi078.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-14T11:57:21.235 INFO:teuthology.orchestra.run.smithi078.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-14T11:57:21.236 INFO:teuthology.orchestra.run.smithi078.stdout: python3-mako-1.0.6-13.el8.noarch 2023-10-14T11:57:21.236 INFO:teuthology.orchestra.run.smithi078.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-14T11:57:21.236 INFO:teuthology.orchestra.run.smithi078.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-14T11:57:21.236 INFO:teuthology.orchestra.run.smithi078.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-10-14T11:57:21.236 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-10-14T11:57:21.237 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-10-14T11:57:21.237 INFO:teuthology.orchestra.run.smithi078.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-14T11:57:21.237 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-14T11:57:21.237 INFO:teuthology.orchestra.run.smithi078.stdout: python3-portend-2.6-1.el8.noarch 2023-10-14T11:57:21.238 INFO:teuthology.orchestra.run.smithi078.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-14T11:57:21.238 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-10-14T11:57:21.238 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-14T11:57:21.238 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-14T11:57:21.238 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pycparser-2.14-14.el8.noarch 2023-10-14T11:57:21.239 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytz-2017.2-9.el8.noarch 2023-10-14T11:57:21.239 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rados-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:21.239 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rbd-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:21.239 INFO:teuthology.orchestra.run.smithi078.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-14T11:57:21.239 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-14T11:57:21.240 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rgw-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:21.240 INFO:teuthology.orchestra.run.smithi078.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-14T11:57:21.240 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-14T11:57:21.240 INFO:teuthology.orchestra.run.smithi078.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-10-14T11:57:21.240 INFO:teuthology.orchestra.run.smithi078.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-14T11:57:21.241 INFO:teuthology.orchestra.run.smithi078.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-14T11:57:21.241 INFO:teuthology.orchestra.run.smithi078.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-14T11:57:21.241 INFO:teuthology.orchestra.run.smithi078.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-14T11:57:21.241 INFO:teuthology.orchestra.run.smithi078.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-14T11:57:21.241 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-14T11:57:21.241 INFO:teuthology.orchestra.run.smithi078.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-14T11:57:21.242 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-14T11:57:21.242 INFO:teuthology.orchestra.run.smithi078.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-14T11:57:21.242 INFO:teuthology.orchestra.run.smithi078.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-14T11:57:21.242 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-fuse-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-mirror-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-nbd-2:18.0.0-6730.g4ba8a190.el8.x86_64 2023-10-14T11:57:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: socat-1.7.4.1-1.el8.x86_64 2023-10-14T11:57:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: thrift-0.13.0-2.el8.x86_64 2023-10-14T11:57:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: uuid-1.6.2-43.el8.x86_64 2023-10-14T11:57:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-10-14T11:57:21.244 INFO:teuthology.orchestra.run.smithi078.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-14T11:57:21.244 INFO:teuthology.orchestra.run.smithi078.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-10-14T11:57:21.244 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:57:21.245 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-10-14T11:57:21.410 DEBUG:teuthology.parallel:result is None 2023-10-14T11:57:21.411 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0 2023-10-14T11:57:21.631 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-14T11:57:21.655 INFO:teuthology.orchestra.run.smithi078.stdout:18.0.0-6730.g4ba8a190.el8 2023-10-14T11:57:21.655 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6730.g4ba8a190.el8 2023-10-14T11:57:21.656 INFO:teuthology.task.install:The correct ceph version 18.0.0-6730.g4ba8a190 is installed. 2023-10-14T11:57:21.656 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0 2023-10-14T11:57:21.809 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-14T11:57:21.838 INFO:teuthology.orchestra.run.smithi195.stdout:18.0.0-6730.g4ba8a190.el8 2023-10-14T11:57:21.839 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6730.g4ba8a190.el8 2023-10-14T11:57:21.839 INFO:teuthology.task.install:The correct ceph version 18.0.0-6730.g4ba8a190 is installed. 2023-10-14T11:57:21.840 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-14T11:57:21.840 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-14T11:57:21.841 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-14T11:57:21.882 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-14T11:57:21.883 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-14T11:57:21.930 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-14T11:57:21.930 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-14T11:57:21.931 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/daemon-helper 2023-10-14T11:57:21.957 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-14T11:57:22.024 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-14T11:57:22.024 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/daemon-helper 2023-10-14T11:57:22.055 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-14T11:57:22.124 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-14T11:57:22.124 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-14T11:57:22.124 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-14T11:57:22.151 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-14T11:57:22.217 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-14T11:57:22.217 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-14T11:57:22.246 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-14T11:57:22.315 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-14T11:57:22.315 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-14T11:57:22.316 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/stdin-killer 2023-10-14T11:57:22.342 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-14T11:57:22.407 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-14T11:57:22.408 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/stdin-killer 2023-10-14T11:57:22.436 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-14T11:57:22.503 INFO:teuthology.run_tasks:Running task ceph... 2023-10-14T11:57:22.572 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-14T11:57:22.573 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 777 /var/log/ceph 2023-10-14T11:57:22.575 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 777 /var/log/ceph 2023-10-14T11:57:22.602 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-14T11:57:22.603 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-14T11:57:22.641 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-14T11:57:22.669 INFO:tasks.ceph:Creating extra log directories... 2023-10-14T11:57:22.670 DEBUG:teuthology.orchestra.run.smithi078:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-14T11:57:22.707 DEBUG:teuthology.orchestra.run.smithi195:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-14T11:57:22.741 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-10-14T11:57:22.742 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-10-14T11:57:22.742 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-10-14T11:57:22.742 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-14T11:57:22.743 DEBUG:teuthology.orchestra.run.smithi078:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-14T11:57:22.787 DEBUG:teuthology.orchestra.run.smithi078:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-14T11:57:22.935 DEBUG:teuthology.orchestra.run.smithi078:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-10-14T11:57:22.962 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-14T11:57:22.963 DEBUG:teuthology.orchestra.run.smithi195:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-14T11:57:22.979 DEBUG:teuthology.orchestra.run.smithi195:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-14T11:57:23.114 DEBUG:teuthology.orchestra.run.smithi195:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-10-14T11:57:23.146 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-14T11:57:23.146 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'Replacing daemon mds', '\\(MDS_INSUFFICIENT_STANDBY\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MDS_DAMAGE\\)', '\\(FS_DEGRADED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-14T11:57:23.147 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rishabh-2023-10-13_18:04:14-fs-rishabh-2023oct13-testing-default-smithi/7425292', 'branch': 'rishabh-2023oct13', 'description': 'fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}', 'email': None, 'first_in_suite': False, 'job_id': '7425292', 'kernel': {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '47672119149b103ec7cec2cc65c063ccbb001ac7'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'rishabh-2023-10-13_18:04:14-fs-rishabh-2023oct13-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'rishabh-2023oct13'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'Replacing daemon mds', '\\(MDS_INSUFFICIENT_STANDBY\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MDS_DAMAGE\\)', '\\(FS_DEGRADED\\)'], 'sha1': '4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'ceph-fuse': {'disabled': True}, 'install': {'ceph': {'flavor': 'default', 'sha1': '4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0'}}, 'kclient': {'syntax': 'v2'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'rishabh-2023oct13', 'sha1': '4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0'}}, 'owner': 'scheduled_rishabh@teuthology', 'priority': 75, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']], 'seed': 6006, 'sha1': '4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0', 'sleep_before_teardown': 0, 'subset': '402/512', 'suite': 'fs', 'suite_branch': 'rishabh-2023oct13', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi078.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs=', 'smithi195.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '47672119149b103ec7cec2cc65c063ccbb001ac7'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'Replacing daemon mds', '\\(MDS_INSUFFICIENT_STANDBY\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MDS_DAMAGE\\)', '\\(FS_DEGRADED\\)'], 'sha1': '4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0', 'cluster': 'ceph'}}, {'kclient': None}, {'exec': {'mon.a': ['ceph config set mgr mgr/crash/warn_recent_interval 0']}}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_failover']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '8cdab074dcca9a68965bc5a50e9c30b691949723', 'timestamp': '2023-10-13_18:04:14', 'tube': 'smithi', 'user': 'rishabh', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-14T11:57:23.148 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-14T11:57:23.150 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-14T11:57:23.197 DEBUG:teuthology.orchestra.run.smithi078:> sudo install -d -m0777 -- /var/run/ceph 2023-10-14T11:57:23.208 DEBUG:teuthology.orchestra.run.smithi195:> sudo install -d -m0777 -- /var/run/ceph 2023-10-14T11:57:23.264 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-14T11:57:23.264 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/scratch_devs of=/dev/stdout 2023-10-14T11:57:23.290 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-14T11:57:23.290 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_1 2023-10-14T11:57:23.347 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-14T11:57:23.347 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-14T11:57:23.347 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 6h/6d Inode: 392412 Links: 1 2023-10-14T11:57:23.348 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-14T11:57:23.348 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2023-10-14T11:57:23.348 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2023-10-14 11:57:12.106765736 +0000 2023-10-14T11:57:23.349 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2023-10-14 11:54:34.317127711 +0000 2023-10-14T11:57:23.349 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2023-10-14 11:54:34.317127711 +0000 2023-10-14T11:57:23.349 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2023-10-14T11:57:23.350 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-14T11:57:23.412 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2023-10-14T11:57:23.413 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2023-10-14T11:57:23.413 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000249061 s, 2.1 MB/s 2023-10-14T11:57:23.414 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-14T11:57:23.472 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_2 2023-10-14T11:57:23.530 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-14T11:57:23.530 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-14T11:57:23.530 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 6h/6d Inode: 390725 Links: 1 2023-10-14T11:57:23.531 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-14T11:57:23.531 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2023-10-14T11:57:23.531 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2023-10-14 11:57:12.106765736 +0000 2023-10-14T11:57:23.532 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2023-10-14 11:54:34.820119324 +0000 2023-10-14T11:57:23.532 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2023-10-14 11:54:34.820119324 +0000 2023-10-14T11:57:23.532 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2023-10-14T11:57:23.533 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-14T11:57:23.594 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2023-10-14T11:57:23.594 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2023-10-14T11:57:23.594 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000224316 s, 2.3 MB/s 2023-10-14T11:57:23.595 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-14T11:57:23.653 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_3 2023-10-14T11:57:23.709 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-14T11:57:23.710 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-14T11:57:23.710 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 6h/6d Inode: 387832 Links: 1 2023-10-14T11:57:23.710 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-14T11:57:23.710 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2023-10-14T11:57:23.710 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2023-10-14 11:57:12.106765736 +0000 2023-10-14T11:57:23.711 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2023-10-14 11:54:35.323110936 +0000 2023-10-14T11:57:23.711 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2023-10-14 11:54:35.323110936 +0000 2023-10-14T11:57:23.711 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2023-10-14T11:57:23.711 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-14T11:57:23.774 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2023-10-14T11:57:23.774 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2023-10-14T11:57:23.775 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000240925 s, 2.1 MB/s 2023-10-14T11:57:23.776 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-14T11:57:23.833 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_4 2023-10-14T11:57:23.890 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-14T11:57:23.890 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-14T11:57:23.891 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 6h/6d Inode: 390904 Links: 1 2023-10-14T11:57:23.891 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-14T11:57:23.891 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2023-10-14T11:57:23.891 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2023-10-14 11:57:12.106765736 +0000 2023-10-14T11:57:23.891 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2023-10-14 11:54:35.815102732 +0000 2023-10-14T11:57:23.892 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2023-10-14 11:54:35.815102732 +0000 2023-10-14T11:57:23.892 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2023-10-14T11:57:23.892 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-14T11:57:23.955 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2023-10-14T11:57:23.955 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2023-10-14T11:57:23.956 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000186023 s, 2.8 MB/s 2023-10-14T11:57:23.957 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-14T11:57:24.014 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-14T11:57:24.014 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-14T11:57:24.014 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/scratch_devs of=/dev/stdout 2023-10-14T11:57:24.035 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-14T11:57:24.035 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_1 2023-10-14T11:57:24.094 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-14T11:57:24.094 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-14T11:57:24.094 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 6h/6d Inode: 387842 Links: 1 2023-10-14T11:57:24.095 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-14T11:57:24.095 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2023-10-14T11:57:24.095 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2023-10-14 11:57:06.214862264 +0000 2023-10-14T11:57:24.095 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2023-10-14 11:54:33.573786077 +0000 2023-10-14T11:57:24.095 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2023-10-14 11:54:33.573786077 +0000 2023-10-14T11:57:24.096 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: - 2023-10-14T11:57:24.096 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-14T11:57:24.163 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2023-10-14T11:57:24.163 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2023-10-14T11:57:24.163 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000130995 s, 3.9 MB/s 2023-10-14T11:57:24.164 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-14T11:57:24.224 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_2 2023-10-14T11:57:24.283 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-14T11:57:24.283 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-14T11:57:24.283 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 6h/6d Inode: 394375 Links: 1 2023-10-14T11:57:24.284 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-14T11:57:24.284 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2023-10-14T11:57:24.284 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2023-10-14 11:57:06.214862264 +0000 2023-10-14T11:57:24.284 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2023-10-14 11:54:34.073776619 +0000 2023-10-14T11:57:24.284 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2023-10-14 11:54:34.073776619 +0000 2023-10-14T11:57:24.285 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: - 2023-10-14T11:57:24.285 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-14T11:57:24.352 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2023-10-14T11:57:24.353 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2023-10-14T11:57:24.353 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000140347 s, 3.6 MB/s 2023-10-14T11:57:24.354 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-14T11:57:24.415 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_3 2023-10-14T11:57:24.474 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-14T11:57:24.475 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-14T11:57:24.475 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 6h/6d Inode: 393514 Links: 1 2023-10-14T11:57:24.475 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-14T11:57:24.475 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2023-10-14T11:57:24.475 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2023-10-14 11:57:06.214862264 +0000 2023-10-14T11:57:24.476 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2023-10-14 11:54:34.590766840 +0000 2023-10-14T11:57:24.476 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2023-10-14 11:54:34.590766840 +0000 2023-10-14T11:57:24.476 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: - 2023-10-14T11:57:24.476 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-14T11:57:24.541 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2023-10-14T11:57:24.542 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2023-10-14T11:57:24.542 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000101623 s, 5.0 MB/s 2023-10-14T11:57:24.543 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-14T11:57:24.602 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_4 2023-10-14T11:57:24.661 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-14T11:57:24.661 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-14T11:57:24.661 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 6h/6d Inode: 388007 Links: 1 2023-10-14T11:57:24.661 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-14T11:57:24.662 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2023-10-14T11:57:24.662 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2023-10-14 11:57:06.214862264 +0000 2023-10-14T11:57:24.662 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2023-10-14 11:54:35.093757325 +0000 2023-10-14T11:57:24.662 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2023-10-14 11:54:35.093757325 +0000 2023-10-14T11:57:24.662 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: - 2023-10-14T11:57:24.663 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-14T11:57:24.728 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2023-10-14T11:57:24.728 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2023-10-14T11:57:24.729 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000134699 s, 3.8 MB/s 2023-10-14T11:57:24.730 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-14T11:57:24.788 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-14T11:57:24.789 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi078.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi195.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-14T11:57:24.789 INFO:tasks.ceph:Generating config... 2023-10-14T11:57:24.790 INFO:tasks.ceph:[client] client mount timeout = 600 2023-10-14T11:57:24.791 INFO:tasks.ceph:[client] debug client = 20 2023-10-14T11:57:24.791 INFO:tasks.ceph:[client] debug ms = 1 2023-10-14T11:57:24.791 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-10-14T11:57:24.792 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-10-14T11:57:24.792 INFO:tasks.ceph:[global] ms die on skipped message = False 2023-10-14T11:57:24.792 INFO:tasks.ceph:[mds] debug mds = 20 2023-10-14T11:57:24.792 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-10-14T11:57:24.793 INFO:tasks.ceph:[mds] debug ms = 1 2023-10-14T11:57:24.793 INFO:tasks.ceph:[mds] mds debug frag = True 2023-10-14T11:57:24.793 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-10-14T11:57:24.793 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-10-14T11:57:24.794 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-10-14T11:57:24.794 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-10-14T11:57:24.794 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-10-14T11:57:24.794 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-10-14T11:57:24.795 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-14T11:57:24.795 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-14T11:57:24.795 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-14T11:57:24.795 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-14T11:57:24.796 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-14T11:57:24.796 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-10-14T11:57:24.796 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-14T11:57:24.796 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-10-14T11:57:24.797 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-14T11:57:24.797 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-10-14T11:57:24.798 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-10-14T11:57:24.798 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-14T11:57:24.802 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-14T11:57:24.802 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-10-14T11:57:24.802 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-14T11:57:24.803 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-14T11:57:24.803 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-14T11:57:24.803 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-14T11:57:24.803 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-14T11:57:24.804 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-10-14T11:57:24.804 INFO:tasks.ceph:Setting up mon.a... 2023-10-14T11:57:24.804 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-14T11:57:24.914 INFO:teuthology.orchestra.run.smithi078.stdout:creating /etc/ceph/ceph.keyring 2023-10-14T11:57:24.917 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-14T11:57:24.967 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-14T11:57:24.994 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.78'), ('mon.b', '172.21.15.195'), ('mon.c', '[v2:172.21.15.195:3301,v1:172.21.15.195:6790]')] 2023-10-14T11:57:24.995 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.78,172.21.15.195,[v2:172.21.15.195:3301,v1:172.21.15.195:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-10-14T11:57:24.996 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-14T11:57:24.996 DEBUG:teuthology.orchestra.run.smithi078:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-14T11:57:25.051 DEBUG:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.78 --add b 172.21.15.195 --addv c '[v2:172.21.15.195:3301,v1:172.21.15.195:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-10-14T11:57:25.139 INFO:teuthology.orchestra.run.smithi078.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-14T11:57:25.139 INFO:teuthology.orchestra.run.smithi078.stdout:monmaptool: generated fsid 4c5dc134-3092-406a-a1e1-13e6d9e926e8 2023-10-14T11:57:25.139 INFO:teuthology.orchestra.run.smithi078.stdout:setting min_mon_release = pacific 2023-10-14T11:57:25.140 INFO:teuthology.orchestra.run.smithi078.stdout:epoch 0 2023-10-14T11:57:25.140 INFO:teuthology.orchestra.run.smithi078.stdout:fsid 4c5dc134-3092-406a-a1e1-13e6d9e926e8 2023-10-14T11:57:25.140 INFO:teuthology.orchestra.run.smithi078.stdout:last_changed 2023-10-14T11:57:25.138487+0000 2023-10-14T11:57:25.140 INFO:teuthology.orchestra.run.smithi078.stdout:created 2023-10-14T11:57:25.138487+0000 2023-10-14T11:57:25.140 INFO:teuthology.orchestra.run.smithi078.stdout:min_mon_release 16 (pacific) 2023-10-14T11:57:25.141 INFO:teuthology.orchestra.run.smithi078.stdout:election_strategy: 1 2023-10-14T11:57:25.141 INFO:teuthology.orchestra.run.smithi078.stdout:0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2023-10-14T11:57:25.141 INFO:teuthology.orchestra.run.smithi078.stdout:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2023-10-14T11:57:25.141 INFO:teuthology.orchestra.run.smithi078.stdout:2: [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] mon.c 2023-10-14T11:57:25.141 INFO:teuthology.orchestra.run.smithi078.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-10-14T11:57:25.143 DEBUG:teuthology.orchestra.run.smithi078:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-14T11:57:25.159 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 4c5dc134-3092-406a-a1e1-13e6d9e926e8... 2023-10-14T11:57:25.160 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-14T11:57:25.202 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-14T11:57:25.244 INFO:teuthology.orchestra.run.smithi078.stdout:[global] 2023-10-14T11:57:25.244 INFO:teuthology.orchestra.run.smithi078.stdout: chdir = "" 2023-10-14T11:57:25.245 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-14T11:57:25.245 INFO:teuthology.orchestra.run.smithi078.stdout: auth supported = cephx 2023-10-14T11:57:25.245 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:57:25.245 INFO:teuthology.orchestra.run.smithi078.stdout: filestore xattr use omap = true 2023-10-14T11:57:25.245 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:57:25.246 INFO:teuthology.orchestra.run.smithi078.stdout: mon clock drift allowed = 1.000 2023-10-14T11:57:25.246 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:57:25.246 INFO:teuthology.orchestra.run.smithi078.stdout: osd crush chooseleaf type = 0 2023-10-14T11:57:25.246 INFO:teuthology.orchestra.run.smithi078.stdout: auth debug = true 2023-10-14T11:57:25.246 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:57:25.247 INFO:teuthology.orchestra.run.smithi078.stdout: ms die on old message = true 2023-10-14T11:57:25.247 INFO:teuthology.orchestra.run.smithi078.stdout: ms die on bug = true 2023-10-14T11:57:25.247 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:57:25.247 INFO:teuthology.orchestra.run.smithi078.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-14T11:57:25.247 INFO:teuthology.orchestra.run.smithi078.stdout: mon pg warn max object skew = 0 2023-10-14T11:57:25.247 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:57:25.248 INFO:teuthology.orchestra.run.smithi078.stdout: # disable pg_autoscaler by default for new pools 2023-10-14T11:57:25.248 INFO:teuthology.orchestra.run.smithi078.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-14T11:57:25.248 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:57:25.248 INFO:teuthology.orchestra.run.smithi078.stdout: osd pool default size = 2 2023-10-14T11:57:25.249 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:57:25.249 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd allow primary affinity = true 2023-10-14T11:57:25.249 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd allow pg remap = true 2023-10-14T11:57:25.249 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on legacy crush tunables = false 2023-10-14T11:57:25.249 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on crush straw calc version zero = false 2023-10-14T11:57:25.250 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on no sortbitwise = false 2023-10-14T11:57:25.250 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on osd down out interval zero = false 2023-10-14T11:57:25.250 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on too few osds = false 2023-10-14T11:57:25.250 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-14T11:57:25.251 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-14T11:57:25.251 INFO:teuthology.orchestra.run.smithi078.stdout: mon_allow_pool_size_one = true 2023-10-14T11:57:25.251 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:57:25.251 INFO:teuthology.orchestra.run.smithi078.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-14T11:57:25.251 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:57:25.251 INFO:teuthology.orchestra.run.smithi078.stdout: osd default data pool replay window = 5 2023-10-14T11:57:25.252 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:57:25.252 INFO:teuthology.orchestra.run.smithi078.stdout: mon allow pool delete = true 2023-10-14T11:57:25.252 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:57:25.252 INFO:teuthology.orchestra.run.smithi078.stdout: mon cluster log file level = debug 2023-10-14T11:57:25.252 INFO:teuthology.orchestra.run.smithi078.stdout: debug asserts on shutdown = true 2023-10-14T11:57:25.253 INFO:teuthology.orchestra.run.smithi078.stdout: mon health detail to clog = false 2023-10-14T11:57:25.253 INFO:teuthology.orchestra.run.smithi078.stdout: mon host = "172.21.15.78,172.21.15.195,[v2:172.21.15.195:3301,v1:172.21.15.195:6790]" 2023-10-14T11:57:25.253 INFO:teuthology.orchestra.run.smithi078.stdout: ms die on skipped message = False 2023-10-14T11:57:25.253 INFO:teuthology.orchestra.run.smithi078.stdout: fsid = 4c5dc134-3092-406a-a1e1-13e6d9e926e8 2023-10-14T11:57:25.253 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:57:25.254 INFO:teuthology.orchestra.run.smithi078.stdout:[osd] 2023-10-14T11:57:25.254 INFO:teuthology.orchestra.run.smithi078.stdout: osd journal size = 100 2023-10-14T11:57:25.254 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:57:25.254 INFO:teuthology.orchestra.run.smithi078.stdout: osd scrub load threshold = 5.0 2023-10-14T11:57:25.255 INFO:teuthology.orchestra.run.smithi078.stdout: osd scrub max interval = 600 2023-10-14T11:57:25.255 INFO:teuthology.orchestra.run.smithi078.stdout: osd mclock profile = high_recovery_ops 2023-10-14T11:57:25.255 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:57:25.255 INFO:teuthology.orchestra.run.smithi078.stdout: osd recover clone overlap = true 2023-10-14T11:57:25.255 INFO:teuthology.orchestra.run.smithi078.stdout: osd recovery max chunk = 1048576 2023-10-14T11:57:25.256 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:57:25.256 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug shutdown = true 2023-10-14T11:57:25.256 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug op order = true 2023-10-14T11:57:25.256 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug verify stray on activate = true 2023-10-14T11:57:25.256 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:57:25.257 INFO:teuthology.orchestra.run.smithi078.stdout: osd open classes on start = true 2023-10-14T11:57:25.257 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug pg log writeout = true 2023-10-14T11:57:25.257 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:57:25.257 INFO:teuthology.orchestra.run.smithi078.stdout: osd deep scrub update digest min age = 30 2023-10-14T11:57:25.257 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:57:25.258 INFO:teuthology.orchestra.run.smithi078.stdout: osd map max advance = 10 2023-10-14T11:57:25.258 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:57:25.258 INFO:teuthology.orchestra.run.smithi078.stdout: journal zero on create = true 2023-10-14T11:57:25.259 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:57:25.259 INFO:teuthology.orchestra.run.smithi078.stdout: filestore ondisk finisher threads = 3 2023-10-14T11:57:25.259 INFO:teuthology.orchestra.run.smithi078.stdout: filestore apply finisher threads = 3 2023-10-14T11:57:25.260 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:57:25.260 INFO:teuthology.orchestra.run.smithi078.stdout: bdev debug aio = true 2023-10-14T11:57:25.260 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug misdirected ops = true 2023-10-14T11:57:25.261 INFO:teuthology.orchestra.run.smithi078.stdout: bluestore block size = 96636764160 2023-10-14T11:57:25.261 INFO:teuthology.orchestra.run.smithi078.stdout: bluestore compression mode = aggressive 2023-10-14T11:57:25.261 INFO:teuthology.orchestra.run.smithi078.stdout: bluestore fsck on mount = True 2023-10-14T11:57:25.261 INFO:teuthology.orchestra.run.smithi078.stdout: debug bluefs = 20 2023-10-14T11:57:25.262 INFO:teuthology.orchestra.run.smithi078.stdout: debug bluestore = 20 2023-10-14T11:57:25.262 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2023-10-14T11:57:25.262 INFO:teuthology.orchestra.run.smithi078.stdout: debug osd = 20 2023-10-14T11:57:25.263 INFO:teuthology.orchestra.run.smithi078.stdout: debug rocksdb = 10 2023-10-14T11:57:25.263 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd backfillfull_ratio = 0.85 2023-10-14T11:57:25.263 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd full ratio = 0.9 2023-10-14T11:57:25.264 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd nearfull ratio = 0.8 2023-10-14T11:57:25.264 INFO:teuthology.orchestra.run.smithi078.stdout: osd failsafe full ratio = 0.95 2023-10-14T11:57:25.264 INFO:teuthology.orchestra.run.smithi078.stdout: osd objectstore = bluestore 2023-10-14T11:57:25.264 INFO:teuthology.orchestra.run.smithi078.stdout: osd op complaint time = 180 2023-10-14T11:57:25.264 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:57:25.265 INFO:teuthology.orchestra.run.smithi078.stdout:[mgr] 2023-10-14T11:57:25.265 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2023-10-14T11:57:25.265 INFO:teuthology.orchestra.run.smithi078.stdout: debug mgr = 20 2023-10-14T11:57:25.265 INFO:teuthology.orchestra.run.smithi078.stdout: debug mon = 20 2023-10-14T11:57:25.265 INFO:teuthology.orchestra.run.smithi078.stdout: debug auth = 20 2023-10-14T11:57:25.266 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min pgs per osd = 4 2023-10-14T11:57:25.266 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min bytes per osd = 10 2023-10-14T11:57:25.266 INFO:teuthology.orchestra.run.smithi078.stdout: mgr/telemetry/nag = false 2023-10-14T11:57:25.266 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:57:25.267 INFO:teuthology.orchestra.run.smithi078.stdout:[mon] 2023-10-14T11:57:25.267 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2023-10-14T11:57:25.267 INFO:teuthology.orchestra.run.smithi078.stdout: debug mon = 20 2023-10-14T11:57:25.267 INFO:teuthology.orchestra.run.smithi078.stdout: debug paxos = 20 2023-10-14T11:57:25.267 INFO:teuthology.orchestra.run.smithi078.stdout: debug auth = 20 2023-10-14T11:57:25.267 INFO:teuthology.orchestra.run.smithi078.stdout: mon data avail warn = 5 2023-10-14T11:57:25.268 INFO:teuthology.orchestra.run.smithi078.stdout: mon mgr mkfs grace = 240 2023-10-14T11:57:25.268 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min pgs per osd = 4 2023-10-14T11:57:25.268 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd reporter subtree level = osd 2023-10-14T11:57:25.268 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd prime pg temp = true 2023-10-14T11:57:25.269 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min bytes per osd = 10 2023-10-14T11:57:25.269 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:57:25.269 INFO:teuthology.orchestra.run.smithi078.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-14T11:57:25.269 INFO:teuthology.orchestra.run.smithi078.stdout: auth mon ticket ttl = 660 # 11m 2023-10-14T11:57:25.269 INFO:teuthology.orchestra.run.smithi078.stdout: auth service ticket ttl = 240 # 4m 2023-10-14T11:57:25.269 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:57:25.270 INFO:teuthology.orchestra.run.smithi078.stdout: # don't complain about insecure global_id in the test suite 2023-10-14T11:57:25.270 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-14T11:57:25.270 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-14T11:57:25.270 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:57:25.270 INFO:teuthology.orchestra.run.smithi078.stdout: # 1m isn't quite enough 2023-10-14T11:57:25.271 INFO:teuthology.orchestra.run.smithi078.stdout: mon_down_mkfs_grace = 2m 2023-10-14T11:57:25.271 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:57:25.271 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_filestore_osds = false 2023-10-14T11:57:25.271 INFO:teuthology.orchestra.run.smithi078.stdout: mon op complaint time = 120 2023-10-14T11:57:25.271 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:57:25.272 INFO:teuthology.orchestra.run.smithi078.stdout:[client] 2023-10-14T11:57:25.272 INFO:teuthology.orchestra.run.smithi078.stdout: rgw cache enabled = true 2023-10-14T11:57:25.272 INFO:teuthology.orchestra.run.smithi078.stdout: rgw enable ops log = true 2023-10-14T11:57:25.272 INFO:teuthology.orchestra.run.smithi078.stdout: rgw enable usage log = true 2023-10-14T11:57:25.272 INFO:teuthology.orchestra.run.smithi078.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-14T11:57:25.272 INFO:teuthology.orchestra.run.smithi078.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-14T11:57:25.273 INFO:teuthology.orchestra.run.smithi078.stdout: client mount timeout = 600 2023-10-14T11:57:25.273 INFO:teuthology.orchestra.run.smithi078.stdout: debug client = 20 2023-10-14T11:57:25.273 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2023-10-14T11:57:25.273 INFO:teuthology.orchestra.run.smithi078.stdout: rados mon op timeout = 900 2023-10-14T11:57:25.273 INFO:teuthology.orchestra.run.smithi078.stdout: rados osd op timeout = 900 2023-10-14T11:57:25.274 INFO:teuthology.orchestra.run.smithi078.stdout:[mon.a] 2023-10-14T11:57:25.274 INFO:teuthology.orchestra.run.smithi078.stdout:[mon.b] 2023-10-14T11:57:25.274 INFO:teuthology.orchestra.run.smithi078.stdout:[mon.c] 2023-10-14T11:57:25.274 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.a] 2023-10-14T11:57:25.274 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.c] 2023-10-14T11:57:25.274 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.b] 2023-10-14T11:57:25.275 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.d] 2023-10-14T11:57:25.275 INFO:teuthology.orchestra.run.smithi078.stdout:[mds] 2023-10-14T11:57:25.275 INFO:teuthology.orchestra.run.smithi078.stdout: debug mds = 20 2023-10-14T11:57:25.275 INFO:teuthology.orchestra.run.smithi078.stdout: debug mds balancer = 20 2023-10-14T11:57:25.275 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2023-10-14T11:57:25.276 INFO:teuthology.orchestra.run.smithi078.stdout: mds debug frag = True 2023-10-14T11:57:25.276 INFO:teuthology.orchestra.run.smithi078.stdout: mds debug scatterstat = True 2023-10-14T11:57:25.276 INFO:teuthology.orchestra.run.smithi078.stdout: mds op complaint time = 180 2023-10-14T11:57:25.276 INFO:teuthology.orchestra.run.smithi078.stdout: mds verify scatter = True 2023-10-14T11:57:25.276 INFO:teuthology.orchestra.run.smithi078.stdout: osd op complaint time = 180 2023-10-14T11:57:25.276 INFO:teuthology.orchestra.run.smithi078.stdout: rados mon op timeout = 900 2023-10-14T11:57:25.277 INFO:teuthology.orchestra.run.smithi078.stdout: rados osd op timeout = 900 2023-10-14T11:57:25.278 INFO:teuthology.orchestra.run.smithi195.stdout:[global] 2023-10-14T11:57:25.278 INFO:teuthology.orchestra.run.smithi195.stdout: chdir = "" 2023-10-14T11:57:25.279 INFO:teuthology.orchestra.run.smithi195.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-14T11:57:25.279 INFO:teuthology.orchestra.run.smithi195.stdout: auth supported = cephx 2023-10-14T11:57:25.279 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-14T11:57:25.279 INFO:teuthology.orchestra.run.smithi195.stdout: filestore xattr use omap = true 2023-10-14T11:57:25.279 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-14T11:57:25.280 INFO:teuthology.orchestra.run.smithi195.stdout: mon clock drift allowed = 1.000 2023-10-14T11:57:25.280 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-14T11:57:25.280 INFO:teuthology.orchestra.run.smithi195.stdout: osd crush chooseleaf type = 0 2023-10-14T11:57:25.280 INFO:teuthology.orchestra.run.smithi195.stdout: auth debug = true 2023-10-14T11:57:25.280 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-14T11:57:25.281 INFO:teuthology.orchestra.run.smithi195.stdout: ms die on old message = true 2023-10-14T11:57:25.281 INFO:teuthology.orchestra.run.smithi195.stdout: ms die on bug = true 2023-10-14T11:57:25.281 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-14T11:57:25.281 INFO:teuthology.orchestra.run.smithi195.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-14T11:57:25.281 INFO:teuthology.orchestra.run.smithi195.stdout: mon pg warn max object skew = 0 2023-10-14T11:57:25.282 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-14T11:57:25.282 INFO:teuthology.orchestra.run.smithi195.stdout: # disable pg_autoscaler by default for new pools 2023-10-14T11:57:25.282 INFO:teuthology.orchestra.run.smithi195.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-14T11:57:25.282 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-14T11:57:25.282 INFO:teuthology.orchestra.run.smithi195.stdout: osd pool default size = 2 2023-10-14T11:57:25.283 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-14T11:57:25.283 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd allow primary affinity = true 2023-10-14T11:57:25.283 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd allow pg remap = true 2023-10-14T11:57:25.283 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on legacy crush tunables = false 2023-10-14T11:57:25.283 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on crush straw calc version zero = false 2023-10-14T11:57:25.283 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on no sortbitwise = false 2023-10-14T11:57:25.284 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on osd down out interval zero = false 2023-10-14T11:57:25.284 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on too few osds = false 2023-10-14T11:57:25.284 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-14T11:57:25.284 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-14T11:57:25.284 INFO:teuthology.orchestra.run.smithi195.stdout: mon_allow_pool_size_one = true 2023-10-14T11:57:25.285 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-14T11:57:25.285 INFO:teuthology.orchestra.run.smithi195.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-14T11:57:25.285 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-14T11:57:25.285 INFO:teuthology.orchestra.run.smithi195.stdout: osd default data pool replay window = 5 2023-10-14T11:57:25.285 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-14T11:57:25.286 INFO:teuthology.orchestra.run.smithi195.stdout: mon allow pool delete = true 2023-10-14T11:57:25.286 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-14T11:57:25.286 INFO:teuthology.orchestra.run.smithi195.stdout: mon cluster log file level = debug 2023-10-14T11:57:25.286 INFO:teuthology.orchestra.run.smithi195.stdout: debug asserts on shutdown = true 2023-10-14T11:57:25.286 INFO:teuthology.orchestra.run.smithi195.stdout: mon health detail to clog = false 2023-10-14T11:57:25.286 INFO:teuthology.orchestra.run.smithi195.stdout: mon host = "172.21.15.78,172.21.15.195,[v2:172.21.15.195:3301,v1:172.21.15.195:6790]" 2023-10-14T11:57:25.287 INFO:teuthology.orchestra.run.smithi195.stdout: ms die on skipped message = False 2023-10-14T11:57:25.287 INFO:teuthology.orchestra.run.smithi195.stdout: fsid = 4c5dc134-3092-406a-a1e1-13e6d9e926e8 2023-10-14T11:57:25.287 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-14T11:57:25.287 INFO:teuthology.orchestra.run.smithi195.stdout:[osd] 2023-10-14T11:57:25.287 INFO:teuthology.orchestra.run.smithi195.stdout: osd journal size = 100 2023-10-14T11:57:25.288 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-14T11:57:25.288 INFO:teuthology.orchestra.run.smithi195.stdout: osd scrub load threshold = 5.0 2023-10-14T11:57:25.288 INFO:teuthology.orchestra.run.smithi195.stdout: osd scrub max interval = 600 2023-10-14T11:57:25.288 INFO:teuthology.orchestra.run.smithi195.stdout: osd mclock profile = high_recovery_ops 2023-10-14T11:57:25.288 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-14T11:57:25.289 INFO:teuthology.orchestra.run.smithi195.stdout: osd recover clone overlap = true 2023-10-14T11:57:25.289 INFO:teuthology.orchestra.run.smithi195.stdout: osd recovery max chunk = 1048576 2023-10-14T11:57:25.289 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-14T11:57:25.289 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug shutdown = true 2023-10-14T11:57:25.289 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug op order = true 2023-10-14T11:57:25.289 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug verify stray on activate = true 2023-10-14T11:57:25.290 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-14T11:57:25.290 INFO:teuthology.orchestra.run.smithi195.stdout: osd open classes on start = true 2023-10-14T11:57:25.290 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug pg log writeout = true 2023-10-14T11:57:25.290 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-14T11:57:25.290 INFO:teuthology.orchestra.run.smithi195.stdout: osd deep scrub update digest min age = 30 2023-10-14T11:57:25.291 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-14T11:57:25.291 INFO:teuthology.orchestra.run.smithi195.stdout: osd map max advance = 10 2023-10-14T11:57:25.291 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-14T11:57:25.291 INFO:teuthology.orchestra.run.smithi195.stdout: journal zero on create = true 2023-10-14T11:57:25.291 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-14T11:57:25.292 INFO:teuthology.orchestra.run.smithi195.stdout: filestore ondisk finisher threads = 3 2023-10-14T11:57:25.292 INFO:teuthology.orchestra.run.smithi195.stdout: filestore apply finisher threads = 3 2023-10-14T11:57:25.292 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-14T11:57:25.292 INFO:teuthology.orchestra.run.smithi195.stdout: bdev debug aio = true 2023-10-14T11:57:25.292 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug misdirected ops = true 2023-10-14T11:57:25.292 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore block size = 96636764160 2023-10-14T11:57:25.293 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore compression mode = aggressive 2023-10-14T11:57:25.293 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore fsck on mount = True 2023-10-14T11:57:25.293 INFO:teuthology.orchestra.run.smithi195.stdout: debug bluefs = 20 2023-10-14T11:57:25.294 INFO:teuthology.orchestra.run.smithi195.stdout: debug bluestore = 20 2023-10-14T11:57:25.294 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2023-10-14T11:57:25.294 INFO:teuthology.orchestra.run.smithi195.stdout: debug osd = 20 2023-10-14T11:57:25.294 INFO:teuthology.orchestra.run.smithi195.stdout: debug rocksdb = 10 2023-10-14T11:57:25.295 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd backfillfull_ratio = 0.85 2023-10-14T11:57:25.295 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd full ratio = 0.9 2023-10-14T11:57:25.295 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd nearfull ratio = 0.8 2023-10-14T11:57:25.295 INFO:teuthology.orchestra.run.smithi195.stdout: osd failsafe full ratio = 0.95 2023-10-14T11:57:25.295 INFO:teuthology.orchestra.run.smithi195.stdout: osd objectstore = bluestore 2023-10-14T11:57:25.295 INFO:teuthology.orchestra.run.smithi195.stdout: osd op complaint time = 180 2023-10-14T11:57:25.296 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-14T11:57:25.296 INFO:teuthology.orchestra.run.smithi195.stdout:[mgr] 2023-10-14T11:57:25.296 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2023-10-14T11:57:25.296 INFO:teuthology.orchestra.run.smithi195.stdout: debug mgr = 20 2023-10-14T11:57:25.296 INFO:teuthology.orchestra.run.smithi195.stdout: debug mon = 20 2023-10-14T11:57:25.297 INFO:teuthology.orchestra.run.smithi195.stdout: debug auth = 20 2023-10-14T11:57:25.297 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min pgs per osd = 4 2023-10-14T11:57:25.297 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min bytes per osd = 10 2023-10-14T11:57:25.297 INFO:teuthology.orchestra.run.smithi195.stdout: mgr/telemetry/nag = false 2023-10-14T11:57:25.297 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-14T11:57:25.298 INFO:teuthology.orchestra.run.smithi195.stdout:[mon] 2023-10-14T11:57:25.298 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2023-10-14T11:57:25.298 INFO:teuthology.orchestra.run.smithi195.stdout: debug mon = 20 2023-10-14T11:57:25.298 INFO:teuthology.orchestra.run.smithi195.stdout: debug paxos = 20 2023-10-14T11:57:25.298 INFO:teuthology.orchestra.run.smithi195.stdout: debug auth = 20 2023-10-14T11:57:25.299 INFO:teuthology.orchestra.run.smithi195.stdout: mon data avail warn = 5 2023-10-14T11:57:25.299 INFO:teuthology.orchestra.run.smithi195.stdout: mon mgr mkfs grace = 240 2023-10-14T11:57:25.299 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min pgs per osd = 4 2023-10-14T11:57:25.299 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd reporter subtree level = osd 2023-10-14T11:57:25.299 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd prime pg temp = true 2023-10-14T11:57:25.299 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min bytes per osd = 10 2023-10-14T11:57:25.300 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-14T11:57:25.300 INFO:teuthology.orchestra.run.smithi195.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-14T11:57:25.300 INFO:teuthology.orchestra.run.smithi195.stdout: auth mon ticket ttl = 660 # 11m 2023-10-14T11:57:25.300 INFO:teuthology.orchestra.run.smithi195.stdout: auth service ticket ttl = 240 # 4m 2023-10-14T11:57:25.300 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-14T11:57:25.301 INFO:teuthology.orchestra.run.smithi195.stdout: # don't complain about insecure global_id in the test suite 2023-10-14T11:57:25.301 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-14T11:57:25.301 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-14T11:57:25.301 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-14T11:57:25.301 INFO:teuthology.orchestra.run.smithi195.stdout: # 1m isn't quite enough 2023-10-14T11:57:25.301 INFO:teuthology.orchestra.run.smithi195.stdout: mon_down_mkfs_grace = 2m 2023-10-14T11:57:25.302 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-14T11:57:25.302 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_filestore_osds = false 2023-10-14T11:57:25.302 INFO:teuthology.orchestra.run.smithi195.stdout: mon op complaint time = 120 2023-10-14T11:57:25.302 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-14T11:57:25.303 INFO:teuthology.orchestra.run.smithi195.stdout:[client] 2023-10-14T11:57:25.303 INFO:teuthology.orchestra.run.smithi195.stdout: rgw cache enabled = true 2023-10-14T11:57:25.303 INFO:teuthology.orchestra.run.smithi195.stdout: rgw enable ops log = true 2023-10-14T11:57:25.303 INFO:teuthology.orchestra.run.smithi195.stdout: rgw enable usage log = true 2023-10-14T11:57:25.303 INFO:teuthology.orchestra.run.smithi195.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-14T11:57:25.303 INFO:teuthology.orchestra.run.smithi195.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-14T11:57:25.304 INFO:teuthology.orchestra.run.smithi195.stdout: client mount timeout = 600 2023-10-14T11:57:25.304 INFO:teuthology.orchestra.run.smithi195.stdout: debug client = 20 2023-10-14T11:57:25.304 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2023-10-14T11:57:25.304 INFO:teuthology.orchestra.run.smithi195.stdout: rados mon op timeout = 900 2023-10-14T11:57:25.304 INFO:teuthology.orchestra.run.smithi195.stdout: rados osd op timeout = 900 2023-10-14T11:57:25.305 INFO:teuthology.orchestra.run.smithi195.stdout:[mon.a] 2023-10-14T11:57:25.305 INFO:teuthology.orchestra.run.smithi195.stdout:[mon.b] 2023-10-14T11:57:25.305 INFO:teuthology.orchestra.run.smithi195.stdout:[mon.c] 2023-10-14T11:57:25.305 INFO:teuthology.orchestra.run.smithi195.stdout:[mds.a] 2023-10-14T11:57:25.305 INFO:teuthology.orchestra.run.smithi195.stdout:[mds.c] 2023-10-14T11:57:25.305 INFO:teuthology.orchestra.run.smithi195.stdout:[mds.b] 2023-10-14T11:57:25.306 INFO:teuthology.orchestra.run.smithi195.stdout:[mds.d] 2023-10-14T11:57:25.306 INFO:teuthology.orchestra.run.smithi195.stdout:[mds] 2023-10-14T11:57:25.306 INFO:teuthology.orchestra.run.smithi195.stdout: debug mds = 20 2023-10-14T11:57:25.306 INFO:teuthology.orchestra.run.smithi195.stdout: debug mds balancer = 20 2023-10-14T11:57:25.307 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2023-10-14T11:57:25.307 INFO:teuthology.orchestra.run.smithi195.stdout: mds debug frag = True 2023-10-14T11:57:25.307 INFO:teuthology.orchestra.run.smithi195.stdout: mds debug scatterstat = True 2023-10-14T11:57:25.307 INFO:teuthology.orchestra.run.smithi195.stdout: mds op complaint time = 180 2023-10-14T11:57:25.308 INFO:teuthology.orchestra.run.smithi195.stdout: mds verify scatter = True 2023-10-14T11:57:25.308 INFO:teuthology.orchestra.run.smithi195.stdout: osd op complaint time = 180 2023-10-14T11:57:25.308 INFO:teuthology.orchestra.run.smithi195.stdout: rados mon op timeout = 900 2023-10-14T11:57:25.308 INFO:teuthology.orchestra.run.smithi195.stdout: rados osd op timeout = 900 2023-10-14T11:57:25.309 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-14T11:57:25.309 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-14T11:57:25.376 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-14T11:57:25.376 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-14T11:57:25.376 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-14T11:57:25.393 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-14T11:57:25.393 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-14T11:57:25.450 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi078.front.sepia.ceph.com 2023-10-14T11:57:25.450 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-14T11:57:25.450 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-14T11:57:25.451 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-14T11:57:25.525 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-14T11:57:25.526 DEBUG:teuthology.orchestra.run.smithi078:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-14T11:57:25.582 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi195.front.sepia.ceph.com 2023-10-14T11:57:25.583 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-14T11:57:25.583 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-14T11:57:25.583 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-14T11:57:25.627 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-14T11:57:25.628 DEBUG:teuthology.orchestra.run.smithi195:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-14T11:57:25.687 INFO:tasks.ceph:Setting up mon nodes... 2023-10-14T11:57:25.687 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-14T11:57:25.687 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-10-14T11:57:25.740 INFO:teuthology.orchestra.run.smithi078.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-10-14T11:57:25.743 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-14T11:57:25.853 INFO:teuthology.orchestra.run.smithi195.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-14T11:57:25.856 INFO:tasks.ceph:Setting up mds nodes... 2023-10-14T11:57:25.857 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-10-14T11:57:25.909 INFO:teuthology.orchestra.run.smithi078.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-10-14T11:57:25.912 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-10-14T11:57:25.938 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-10-14T11:57:26.031 INFO:teuthology.orchestra.run.smithi078.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-10-14T11:57:26.033 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-10-14T11:57:26.060 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-10-14T11:57:26.119 INFO:teuthology.orchestra.run.smithi195.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-10-14T11:57:26.121 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-10-14T11:57:26.149 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-10-14T11:57:26.243 INFO:teuthology.orchestra.run.smithi195.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-10-14T11:57:26.246 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-10-14T11:57:26.272 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-14T11:57:26.273 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-14T11:57:26.317 INFO:teuthology.orchestra.run.smithi078.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-14T11:57:26.329 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-10-14T11:57:26.373 INFO:teuthology.orchestra.run.smithi195.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-10-14T11:57:26.385 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-14T11:57:26.386 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi078.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi195.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-14T11:57:26.386 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-14T11:57:26.413 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-14T11:57:26.413 INFO:tasks.ceph:role: osd.0 2023-10-14T11:57:26.413 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi078.front.sepia.ceph.com 2023-10-14T11:57:26.414 DEBUG:teuthology.orchestra.run.smithi078:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-14T11:57:26.486 INFO:teuthology.orchestra.run.smithi078.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-14T11:57:26.487 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-14T11:57:26.487 INFO:teuthology.orchestra.run.smithi078.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-14T11:57:26.487 INFO:teuthology.orchestra.run.smithi078.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-14T11:57:26.487 INFO:teuthology.orchestra.run.smithi078.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-14T11:57:26.487 INFO:teuthology.orchestra.run.smithi078.stdout: = sunit=0 swidth=0 blks 2023-10-14T11:57:26.488 INFO:teuthology.orchestra.run.smithi078.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-14T11:57:26.488 INFO:teuthology.orchestra.run.smithi078.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-14T11:57:26.488 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-14T11:57:26.488 INFO:teuthology.orchestra.run.smithi078.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-14T11:57:26.914 INFO:teuthology.orchestra.run.smithi078.stdout:Discarding blocks...Done. 2023-10-14T11:57:26.916 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi078.front.sepia.ceph.com -o noatime 2023-10-14T11:57:26.916 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-14T11:57:27.062 DEBUG:teuthology.orchestra.run.smithi078:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-14T11:57:27.098 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-14T11:57:27.165 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-14T11:57:27.166 INFO:tasks.ceph:role: osd.1 2023-10-14T11:57:27.166 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi078.front.sepia.ceph.com 2023-10-14T11:57:27.166 DEBUG:teuthology.orchestra.run.smithi078:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-14T11:57:27.239 INFO:teuthology.orchestra.run.smithi078.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-14T11:57:27.240 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-14T11:57:27.240 INFO:teuthology.orchestra.run.smithi078.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-14T11:57:27.240 INFO:teuthology.orchestra.run.smithi078.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-14T11:57:27.240 INFO:teuthology.orchestra.run.smithi078.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-14T11:57:27.241 INFO:teuthology.orchestra.run.smithi078.stdout: = sunit=0 swidth=0 blks 2023-10-14T11:57:27.241 INFO:teuthology.orchestra.run.smithi078.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-14T11:57:27.241 INFO:teuthology.orchestra.run.smithi078.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-14T11:57:27.241 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-14T11:57:27.241 INFO:teuthology.orchestra.run.smithi078.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-14T11:57:27.659 INFO:teuthology.orchestra.run.smithi078.stdout:Discarding blocks...Done. 2023-10-14T11:57:27.661 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi078.front.sepia.ceph.com -o noatime 2023-10-14T11:57:27.661 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-14T11:57:27.749 DEBUG:teuthology.orchestra.run.smithi078:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-14T11:57:27.782 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-14T11:57:27.850 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-14T11:57:27.850 INFO:tasks.ceph:role: osd.2 2023-10-14T11:57:27.850 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi078.front.sepia.ceph.com 2023-10-14T11:57:27.851 DEBUG:teuthology.orchestra.run.smithi078:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-14T11:57:27.924 INFO:teuthology.orchestra.run.smithi078.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-14T11:57:27.924 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-14T11:57:27.925 INFO:teuthology.orchestra.run.smithi078.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-14T11:57:27.925 INFO:teuthology.orchestra.run.smithi078.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-14T11:57:27.925 INFO:teuthology.orchestra.run.smithi078.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-14T11:57:27.925 INFO:teuthology.orchestra.run.smithi078.stdout: = sunit=0 swidth=0 blks 2023-10-14T11:57:27.925 INFO:teuthology.orchestra.run.smithi078.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-14T11:57:27.926 INFO:teuthology.orchestra.run.smithi078.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-14T11:57:27.926 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-14T11:57:27.926 INFO:teuthology.orchestra.run.smithi078.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-14T11:57:28.370 INFO:teuthology.orchestra.run.smithi078.stdout:Discarding blocks...Done. 2023-10-14T11:57:28.372 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi078.front.sepia.ceph.com -o noatime 2023-10-14T11:57:28.372 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-14T11:57:28.495 DEBUG:teuthology.orchestra.run.smithi078:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-14T11:57:28.532 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-10-14T11:57:28.602 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-14T11:57:28.603 INFO:tasks.ceph:role: osd.3 2023-10-14T11:57:28.603 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi078.front.sepia.ceph.com 2023-10-14T11:57:28.603 DEBUG:teuthology.orchestra.run.smithi078:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-14T11:57:28.681 INFO:teuthology.orchestra.run.smithi078.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-14T11:57:28.681 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-14T11:57:28.682 INFO:teuthology.orchestra.run.smithi078.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-14T11:57:28.682 INFO:teuthology.orchestra.run.smithi078.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-14T11:57:28.682 INFO:teuthology.orchestra.run.smithi078.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-14T11:57:28.682 INFO:teuthology.orchestra.run.smithi078.stdout: = sunit=0 swidth=0 blks 2023-10-14T11:57:28.682 INFO:teuthology.orchestra.run.smithi078.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-14T11:57:28.683 INFO:teuthology.orchestra.run.smithi078.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-14T11:57:28.685 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-14T11:57:28.685 INFO:teuthology.orchestra.run.smithi078.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-14T11:57:29.138 INFO:teuthology.orchestra.run.smithi078.stdout:Discarding blocks...Done. 2023-10-14T11:57:29.140 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi078.front.sepia.ceph.com -o noatime 2023-10-14T11:57:29.140 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-10-14T11:57:29.321 DEBUG:teuthology.orchestra.run.smithi078:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-10-14T11:57:29.358 DEBUG:teuthology.orchestra.run.smithi078:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-14T11:57:29.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:29.461+0000 7f53cdf5d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-14T11:57:29.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:29.462+0000 7f53cdf5d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-14T11:57:29.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:29.462+0000 7f53cdf5d640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-14T11:57:29.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:29.462+0000 7f53cdf5d640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-14T11:57:29.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:29.462+0000 7f53cdf5d640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-14T11:57:29.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:29.463+0000 7f53cdf5d640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-14T11:57:32.333 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-14T11:57:32.365 DEBUG:teuthology.orchestra.run.smithi078:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-14T11:57:32.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:32.459+0000 7feda74eb640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-14T11:57:32.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:32.459+0000 7feda74eb640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-14T11:57:32.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:32.459+0000 7feda74eb640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-14T11:57:32.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:32.459+0000 7feda74eb640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-14T11:57:32.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:32.459+0000 7feda74eb640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-14T11:57:32.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:32.459+0000 7feda74eb640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-14T11:57:35.329 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-14T11:57:35.362 DEBUG:teuthology.orchestra.run.smithi078:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-14T11:57:35.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:35.455+0000 7f01791c8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-14T11:57:35.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:35.455+0000 7f01791c8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-14T11:57:35.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:35.455+0000 7f01791c8640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-14T11:57:35.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:35.455+0000 7f01791c8640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-14T11:57:35.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:35.455+0000 7f01791c8640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-14T11:57:35.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:35.456+0000 7f01791c8640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-14T11:57:38.333 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-14T11:57:38.367 DEBUG:teuthology.orchestra.run.smithi078:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-14T11:57:38.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:38.460+0000 7fc430589640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-10-14T11:57:38.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:38.460+0000 7fc430589640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-10-14T11:57:38.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:38.460+0000 7fc430589640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-14T11:57:38.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:38.460+0000 7fc430589640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-14T11:57:38.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:38.460+0000 7fc430589640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-14T11:57:38.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:38.461+0000 7fc430589640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-10-14T11:57:41.325 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-10-14T11:57:41.360 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-10-14T11:57:41.394 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-14T11:57:41.394 INFO:tasks.ceph:role: osd.4 2023-10-14T11:57:41.395 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi195.front.sepia.ceph.com 2023-10-14T11:57:41.395 DEBUG:teuthology.orchestra.run.smithi195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-14T11:57:41.468 INFO:teuthology.orchestra.run.smithi195.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-14T11:57:41.468 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-14T11:57:41.468 INFO:teuthology.orchestra.run.smithi195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-14T11:57:41.468 INFO:teuthology.orchestra.run.smithi195.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-14T11:57:41.469 INFO:teuthology.orchestra.run.smithi195.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-14T11:57:41.469 INFO:teuthology.orchestra.run.smithi195.stdout: = sunit=0 swidth=0 blks 2023-10-14T11:57:41.469 INFO:teuthology.orchestra.run.smithi195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-14T11:57:41.470 INFO:teuthology.orchestra.run.smithi195.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-14T11:57:41.470 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-14T11:57:41.470 INFO:teuthology.orchestra.run.smithi195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-14T11:57:41.491 INFO:teuthology.orchestra.run.smithi195.stdout:Discarding blocks...Done. 2023-10-14T11:57:41.493 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi195.front.sepia.ceph.com -o noatime 2023-10-14T11:57:41.494 DEBUG:teuthology.orchestra.run.smithi195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-10-14T11:57:41.578 DEBUG:teuthology.orchestra.run.smithi195:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-10-14T11:57:41.650 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-10-14T11:57:41.716 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-14T11:57:41.717 INFO:tasks.ceph:role: osd.5 2023-10-14T11:57:41.717 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi195.front.sepia.ceph.com 2023-10-14T11:57:41.717 DEBUG:teuthology.orchestra.run.smithi195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-14T11:57:41.782 INFO:teuthology.orchestra.run.smithi195.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-14T11:57:41.782 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-14T11:57:41.783 INFO:teuthology.orchestra.run.smithi195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-14T11:57:41.783 INFO:teuthology.orchestra.run.smithi195.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-14T11:57:41.783 INFO:teuthology.orchestra.run.smithi195.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-14T11:57:41.783 INFO:teuthology.orchestra.run.smithi195.stdout: = sunit=0 swidth=0 blks 2023-10-14T11:57:41.783 INFO:teuthology.orchestra.run.smithi195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-14T11:57:41.784 INFO:teuthology.orchestra.run.smithi195.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-14T11:57:41.784 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-14T11:57:41.784 INFO:teuthology.orchestra.run.smithi195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-14T11:57:41.805 INFO:teuthology.orchestra.run.smithi195.stdout:Discarding blocks...Done. 2023-10-14T11:57:41.807 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi195.front.sepia.ceph.com -o noatime 2023-10-14T11:57:41.807 DEBUG:teuthology.orchestra.run.smithi195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-10-14T11:57:41.886 DEBUG:teuthology.orchestra.run.smithi195:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-10-14T11:57:41.955 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-10-14T11:57:42.022 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-14T11:57:42.022 INFO:tasks.ceph:role: osd.6 2023-10-14T11:57:42.022 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi195.front.sepia.ceph.com 2023-10-14T11:57:42.023 DEBUG:teuthology.orchestra.run.smithi195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-14T11:57:42.087 INFO:teuthology.orchestra.run.smithi195.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-14T11:57:42.088 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-14T11:57:42.088 INFO:teuthology.orchestra.run.smithi195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-14T11:57:42.088 INFO:teuthology.orchestra.run.smithi195.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-14T11:57:42.088 INFO:teuthology.orchestra.run.smithi195.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-14T11:57:42.088 INFO:teuthology.orchestra.run.smithi195.stdout: = sunit=0 swidth=0 blks 2023-10-14T11:57:42.089 INFO:teuthology.orchestra.run.smithi195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-14T11:57:42.089 INFO:teuthology.orchestra.run.smithi195.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-14T11:57:42.089 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-14T11:57:42.089 INFO:teuthology.orchestra.run.smithi195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-14T11:57:42.110 INFO:teuthology.orchestra.run.smithi195.stdout:Discarding blocks...Done. 2023-10-14T11:57:42.111 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi195.front.sepia.ceph.com -o noatime 2023-10-14T11:57:42.112 DEBUG:teuthology.orchestra.run.smithi195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-10-14T11:57:42.191 DEBUG:teuthology.orchestra.run.smithi195:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-10-14T11:57:42.261 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-10-14T11:57:42.328 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-14T11:57:42.328 INFO:tasks.ceph:role: osd.7 2023-10-14T11:57:42.328 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi195.front.sepia.ceph.com 2023-10-14T11:57:42.329 DEBUG:teuthology.orchestra.run.smithi195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-14T11:57:42.393 INFO:teuthology.orchestra.run.smithi195.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-14T11:57:42.393 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-14T11:57:42.393 INFO:teuthology.orchestra.run.smithi195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-14T11:57:42.394 INFO:teuthology.orchestra.run.smithi195.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-14T11:57:42.394 INFO:teuthology.orchestra.run.smithi195.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-14T11:57:42.394 INFO:teuthology.orchestra.run.smithi195.stdout: = sunit=0 swidth=0 blks 2023-10-14T11:57:42.394 INFO:teuthology.orchestra.run.smithi195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-14T11:57:42.394 INFO:teuthology.orchestra.run.smithi195.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-14T11:57:42.395 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-14T11:57:42.395 INFO:teuthology.orchestra.run.smithi195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-14T11:57:42.415 INFO:teuthology.orchestra.run.smithi195.stdout:Discarding blocks...Done. 2023-10-14T11:57:42.417 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi195.front.sepia.ceph.com -o noatime 2023-10-14T11:57:42.417 DEBUG:teuthology.orchestra.run.smithi195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-10-14T11:57:42.497 DEBUG:teuthology.orchestra.run.smithi195:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-10-14T11:57:42.566 DEBUG:teuthology.orchestra.run.smithi195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-14T11:57:42.669 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:42.667+0000 7f0b45a35640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-10-14T11:57:42.669 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:42.667+0000 7f0b45a35640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-10-14T11:57:42.670 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:42.667+0000 7f0b45a35640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-14T11:57:42.670 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:42.667+0000 7f0b45a35640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-14T11:57:42.670 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:42.667+0000 7f0b45a35640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-14T11:57:42.670 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:42.667+0000 7f0b45a35640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-10-14T11:57:45.524 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-10-14T11:57:45.552 DEBUG:teuthology.orchestra.run.smithi195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-14T11:57:45.639 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:45.637+0000 7f191755d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-10-14T11:57:45.639 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:45.637+0000 7f191755d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-10-14T11:57:45.640 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:45.637+0000 7f191755d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-14T11:57:45.640 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:45.637+0000 7f191755d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-14T11:57:45.640 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:45.637+0000 7f191755d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-14T11:57:45.640 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:45.637+0000 7f191755d640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-10-14T11:57:48.501 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-10-14T11:57:48.533 DEBUG:teuthology.orchestra.run.smithi195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-14T11:57:48.625 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:48.624+0000 7f63ba610640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-10-14T11:57:48.626 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:48.624+0000 7f63ba610640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-10-14T11:57:48.626 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:48.624+0000 7f63ba610640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-14T11:57:48.626 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:48.624+0000 7f63ba610640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-14T11:57:48.627 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:48.624+0000 7f63ba610640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-14T11:57:48.627 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:48.625+0000 7f63ba610640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-10-14T11:57:51.493 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-10-14T11:57:51.524 DEBUG:teuthology.orchestra.run.smithi195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-14T11:57:51.618 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:51.616+0000 7fce4f124640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-10-14T11:57:51.618 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:51.617+0000 7fce4f124640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-10-14T11:57:51.618 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:51.617+0000 7fce4f124640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-14T11:57:51.618 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:51.617+0000 7fce4f124640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-14T11:57:51.619 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:51.617+0000 7fce4f124640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-14T11:57:51.619 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:51.617+0000 7fce4f124640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-10-14T11:57:53.150 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T11:57:53.153 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T11:57:54.484 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-10-14T11:57:54.516 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-14T11:57:54.517 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-14T11:57:54.517 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-10-14T11:57:54.547 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-14T11:57:54.548 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-10-14T11:57:54.616 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-14T11:57:54.617 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-10-14T11:57:54.685 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-14T11:57:54.685 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-14T11:57:54.752 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-14T11:57:54.752 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-14T11:57:54.818 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-14T11:57:54.819 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-14T11:57:54.885 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-14T11:57:54.885 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-10-14T11:57:54.950 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-14T11:57:54.951 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-14T11:57:54.982 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-14T11:57:54.983 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-10-14T11:57:55.053 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-14T11:57:55.053 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-10-14T11:57:55.121 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-14T11:57:55.122 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-10-14T11:57:55.190 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-14T11:57:55.190 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-10-14T11:57:55.258 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-14T11:57:55.259 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-10-14T11:57:55.326 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-14T11:57:55.326 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-10-14T11:57:55.392 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-14T11:57:55.393 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-14T11:57:55.412 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-14T11:57:55.412 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-10-14T11:57:55.449 INFO:tasks.ceph:Adding keys to all mons... 2023-10-14T11:57:55.450 DEBUG:teuthology.orchestra.run.smithi078:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-14T11:57:55.455 DEBUG:teuthology.orchestra.run.smithi195:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-14T11:57:55.492 INFO:teuthology.orchestra.run.smithi078.stdout:[mgr.y] 2023-10-14T11:57:55.493 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQAlgiplUokaLBAAJCZkFm8ccnPzzWOU3olSzw== 2023-10-14T11:57:55.493 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.a] 2023-10-14T11:57:55.493 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQAlgipldx40NhAA5+wjFFtoBDWvyFlYkTz5Rg== 2023-10-14T11:57:55.493 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.c] 2023-10-14T11:57:55.494 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQAmgipli8TXARAAcFS0dWo49wsfm5kaTsXnWg== 2023-10-14T11:57:55.494 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.0] 2023-10-14T11:57:55.494 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQApgiplJwmQGxAAXm76A5lPkboC5Ldi1/sEAw== 2023-10-14T11:57:55.494 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.1] 2023-10-14T11:57:55.494 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQAsgiplhKZuGxAAIGuMc3wnXW5P3hZrGXDYYA== 2023-10-14T11:57:55.495 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.2] 2023-10-14T11:57:55.495 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQAvgiplKe43GxAA0V3t/PIav9W0B6QRbGR/WQ== 2023-10-14T11:57:55.495 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.3] 2023-10-14T11:57:55.495 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQAygiplGxOAGxAAThRNVROQeMI4y+sipJ7u1A== 2023-10-14T11:57:55.495 INFO:teuthology.orchestra.run.smithi078.stdout:[mgr.x] 2023-10-14T11:57:55.496 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQAlgiplxL7YMhAAlbcqrnh1wQDuNKCP2XLvFQ== 2023-10-14T11:57:55.496 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.b] 2023-10-14T11:57:55.496 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQAmgiplG9sSBxAAv9SDzXKMbqC9XA8nstj/lQ== 2023-10-14T11:57:55.496 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.d] 2023-10-14T11:57:55.496 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQAmgiplSnGADhAAejccrmfYZiTxuKlX6yuIvQ== 2023-10-14T11:57:55.496 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.4] 2023-10-14T11:57:55.497 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQA2giplqoTWJxAA48WUfi3wb40Z6q8YGsQIBQ== 2023-10-14T11:57:55.497 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.5] 2023-10-14T11:57:55.497 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQA5giplcTEKJhAAN7q3yytrGhq1rdhRohk+4A== 2023-10-14T11:57:55.497 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.6] 2023-10-14T11:57:55.497 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQA8gipl7JU8JRAA0V4vMatkHCoyDp6W1kfB/w== 2023-10-14T11:57:55.498 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.7] 2023-10-14T11:57:55.498 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQA/giplFXfHJBAAafwt2EliCJoj1qSq5utNcQ== 2023-10-14T11:57:55.498 INFO:teuthology.orchestra.run.smithi078.stdout:[client.0] 2023-10-14T11:57:55.498 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQAmgiplorThEhAAXHGu+BK6WV8zJzecocBP5Q== 2023-10-14T11:57:55.499 INFO:teuthology.orchestra.run.smithi078.stdout:[client.1] 2023-10-14T11:57:55.499 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQAmgipl3JM0FhAA2tQCm3Ps6ItAbapH+C46dg== 2023-10-14T11:57:55.515 INFO:teuthology.orchestra.run.smithi195.stdout:[mgr.y] 2023-10-14T11:57:55.515 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAlgiplUokaLBAAJCZkFm8ccnPzzWOU3olSzw== 2023-10-14T11:57:55.515 INFO:teuthology.orchestra.run.smithi195.stdout:[mds.a] 2023-10-14T11:57:55.515 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAlgipldx40NhAA5+wjFFtoBDWvyFlYkTz5Rg== 2023-10-14T11:57:55.516 INFO:teuthology.orchestra.run.smithi195.stdout:[mds.c] 2023-10-14T11:57:55.516 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAmgipli8TXARAAcFS0dWo49wsfm5kaTsXnWg== 2023-10-14T11:57:55.516 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.0] 2023-10-14T11:57:55.516 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQApgiplJwmQGxAAXm76A5lPkboC5Ldi1/sEAw== 2023-10-14T11:57:55.516 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.1] 2023-10-14T11:57:55.517 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAsgiplhKZuGxAAIGuMc3wnXW5P3hZrGXDYYA== 2023-10-14T11:57:55.517 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.2] 2023-10-14T11:57:55.517 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAvgiplKe43GxAA0V3t/PIav9W0B6QRbGR/WQ== 2023-10-14T11:57:55.517 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.3] 2023-10-14T11:57:55.517 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAygiplGxOAGxAAThRNVROQeMI4y+sipJ7u1A== 2023-10-14T11:57:55.517 INFO:teuthology.orchestra.run.smithi195.stdout:[mgr.x] 2023-10-14T11:57:55.518 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAlgiplxL7YMhAAlbcqrnh1wQDuNKCP2XLvFQ== 2023-10-14T11:57:55.518 INFO:teuthology.orchestra.run.smithi195.stdout:[mds.b] 2023-10-14T11:57:55.518 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAmgiplG9sSBxAAv9SDzXKMbqC9XA8nstj/lQ== 2023-10-14T11:57:55.518 INFO:teuthology.orchestra.run.smithi195.stdout:[mds.d] 2023-10-14T11:57:55.518 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAmgiplSnGADhAAejccrmfYZiTxuKlX6yuIvQ== 2023-10-14T11:57:55.519 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.4] 2023-10-14T11:57:55.519 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQA2giplqoTWJxAA48WUfi3wb40Z6q8YGsQIBQ== 2023-10-14T11:57:55.519 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.5] 2023-10-14T11:57:55.519 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQA5giplcTEKJhAAN7q3yytrGhq1rdhRohk+4A== 2023-10-14T11:57:55.519 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.6] 2023-10-14T11:57:55.520 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQA8gipl7JU8JRAA0V4vMatkHCoyDp6W1kfB/w== 2023-10-14T11:57:55.520 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.7] 2023-10-14T11:57:55.520 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQA/giplFXfHJBAAafwt2EliCJoj1qSq5utNcQ== 2023-10-14T11:57:55.520 INFO:teuthology.orchestra.run.smithi195.stdout:[client.0] 2023-10-14T11:57:55.520 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAmgiplorThEhAAXHGu+BK6WV8zJzecocBP5Q== 2023-10-14T11:57:55.521 INFO:teuthology.orchestra.run.smithi195.stdout:[client.1] 2023-10-14T11:57:55.521 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAmgipl3JM0FhAA2tQCm3Ps6ItAbapH+C46dg== 2023-10-14T11:57:55.522 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-14T11:57:55.542 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-14T11:57:55.615 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-14T11:57:55.618 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-14T11:57:55.674 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-14T11:57:55.676 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-14T11:57:55.732 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T11:57:55.735 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T11:57:55.790 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T11:57:55.792 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T11:57:55.846 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T11:57:55.848 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T11:57:55.901 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T11:57:55.904 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T11:57:55.957 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-14T11:57:55.959 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-14T11:57:56.012 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-14T11:57:56.015 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-14T11:57:56.072 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-14T11:57:56.074 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-14T11:57:56.130 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T11:57:56.133 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T11:57:56.187 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T11:57:56.190 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T11:57:56.243 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T11:57:56.246 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T11:57:56.300 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T11:57:56.302 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T11:57:56.355 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-14T11:57:56.358 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-14T11:57:56.414 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-14T11:57:56.416 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-14T11:57:56.472 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-14T11:57:56.473 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-14T11:57:56.499 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-14T11:57:56.597 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-14T11:57:56.624 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-10-14T11:57:56.651 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-14T11:57:56.750 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-10-14T11:57:56.777 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-10-14T11:57:56.844 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-14T11:57:56.941 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-10-14T11:57:56.968 DEBUG:teuthology.orchestra.run.smithi078:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-14T11:57:56.970 DEBUG:teuthology.orchestra.run.smithi195:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-14T11:57:57.025 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-14T11:57:57.025 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-14T11:57:57.026 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-14T11:57:57.028 INFO:tasks.ceph.mon.a:Started 2023-10-14T11:57:57.029 INFO:tasks.ceph.mon.b:Restarting daemon 2023-10-14T11:57:57.029 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-10-14T11:57:57.068 INFO:tasks.ceph.mon.b:Started 2023-10-14T11:57:57.068 INFO:tasks.ceph.mon.c:Restarting daemon 2023-10-14T11:57:57.069 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-10-14T11:57:57.070 INFO:tasks.ceph.mon.c:Started 2023-10-14T11:57:57.071 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-14T11:57:57.071 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-10-14T11:57:57.071 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-10-14T11:57:57.074 INFO:tasks.ceph.mgr.y:Started 2023-10-14T11:57:57.074 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-14T11:57:57.074 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-14T11:57:57.076 INFO:tasks.ceph.mgr.x:Started 2023-10-14T11:57:57.077 INFO:tasks.ceph:Setting crush tunables to default 2023-10-14T11:57:57.077 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --cluster ceph osd crush tunables default 2023-10-14T11:57:57.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.198+0000 7f4d73bb2700 1 Processor -- start 2023-10-14T11:57:57.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.198+0000 7f4d73bb2700 1 -- start start 2023-10-14T11:57:57.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.199+0000 7f4d73bb2700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4d6c13dbe0 0x7f4d6c06a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:57.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.199+0000 7f4d73bb2700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4d6c06ae10 0x7f4d6c0c09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:57.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.199+0000 7f4d73bb2700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d6c06ebe0 con 0x7f4d6c0c4540 2023-10-14T11:57:57.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.199+0000 7f4d73bb2700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d6c06d330 con 0x7f4d6c06ae10 2023-10-14T11:57:57.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.199+0000 7f4d73bb2700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6c1558c0 con 0x7f4d6c13dbe0 2023-10-14T11:57:57.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.199+0000 7f4d7214f700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4d6c06ae10 0x7f4d6c0c09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:57.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.199+0000 7f4d7214f700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4d6c06ae10 0x7f4d6c0c09f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43822/0 (socket says 172.21.15.78:43822) 2023-10-14T11:57:57.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.199+0000 7f4d7214f700 1 -- 172.21.15.78:0/473616330 learned_addr learned my addr 172.21.15.78:0/473616330 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:57:57.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.199+0000 7f4d7194e700 1 --2- 172.21.15.78:0/473616330 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4d6c13dbe0 0x7f4d6c06a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:57.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.199+0000 7f4d7214f700 1 -- 172.21.15.78:0/473616330 >> v2:172.21.15.78:3300/0 conn(0x7f4d6c06ae10 msgr2=0x7f4d6c0c09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-14T11:57:57.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.199+0000 7f4d7214f700 1 -- 172.21.15.78:0/473616330 >> v2:172.21.15.78:3300/0 conn(0x7f4d6c06ae10 msgr2=0x7f4d6c0c09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-14T11:57:57.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.199+0000 7f4d7214f700 1 --2- 172.21.15.78:0/473616330 >> v2:172.21.15.78:3300/0 conn(0x7f4d6c06ae10 0x7f4d6c0c09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-14T11:57:57.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.199+0000 7f4d7194e700 1 -- 172.21.15.78:0/473616330 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4d6c13dbe0 msgr2=0x7f4d6c06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-14T11:57:57.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.199+0000 7f4d7194e700 1 -- 172.21.15.78:0/473616330 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4d6c13dbe0 msgr2=0x7f4d6c06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-14T11:57:57.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.199+0000 7f4d7194e700 1 --2- 172.21.15.78:0/473616330 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4d6c13dbe0 0x7f4d6c06a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-14T11:57:57.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.199+0000 7f4d7214f700 1 --2- 172.21.15.78:0/473616330 >> v2:172.21.15.78:3300/0 conn(0x7f4d6c06ae10 0x7f4d6c0c09f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T11:57:57.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.199+0000 7f4d7194e700 1 --2- 172.21.15.78:0/473616330 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4d6c13dbe0 0x7f4d6c06a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T11:57:57.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.372+0000 7f4d7094c700 1 -- 172.21.15.78:0/473616330 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1796789095 0 0) 0x7f4d6c06ebe0 con 0x7f4d6c0c4540 2023-10-14T11:57:57.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.372+0000 7f4d7094c700 1 -- 172.21.15.78:0/473616330 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d580035b0 con 0x7f4d6c0c4540 2023-10-14T11:57:57.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.373+0000 7f4d7094c700 1 -- 172.21.15.78:0/473616330 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 61187730 0 0) 0x7f4d580035b0 con 0x7f4d6c0c4540 2023-10-14T11:57:57.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.373+0000 7f4d7094c700 1 -- 172.21.15.78:0/473616330 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4d6c13dbe0 msgr2=0x7f4d6c06a8d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-14T11:57:57.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.373+0000 7f4d7094c700 1 --2- 172.21.15.78:0/473616330 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4d6c13dbe0 0x7f4d6c06a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:57.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.373+0000 7f4d7094c700 1 -- 172.21.15.78:0/473616330 >> v2:172.21.15.78:3300/0 conn(0x7f4d6c06ae10 msgr2=0x7f4d6c0c09f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-14T11:57:57.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.373+0000 7f4d7094c700 1 --2- 172.21.15.78:0/473616330 >> v2:172.21.15.78:3300/0 conn(0x7f4d6c06ae10 0x7f4d6c0c09f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:57.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.373+0000 7f4d7094c700 1 -- 172.21.15.78:0/473616330 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d6c154b00 con 0x7f4d6c0c4540 2023-10-14T11:57:57.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.373+0000 7f4d7094c700 1 -- 172.21.15.78:0/473616330 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441799917 0 0) 0x7f4d5c001670 con 0x7f4d6c0c4540 2023-10-14T11:57:57.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.373+0000 7f4d7094c700 1 -- 172.21.15.78:0/473616330 >> v1:172.21.15.195:6789/0 conn(0x7f4d6c0c4540 legacy=0x7f4d6c0c02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:57.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.373+0000 7f4d7094c700 1 --2- 172.21.15.78:0/473616330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d58003a00 0x7f4d58003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:57.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.373+0000 7f4d7094c700 1 --2- 172.21.15.78:0/473616330 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d58004850 0x7f4d58004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:57.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.373+0000 7f4d7094c700 1 --2- 172.21.15.78:0/473616330 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4d6c13dbe0 0x7f4d58005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:57.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.373+0000 7f4d7094c700 1 -- 172.21.15.78:0/473616330 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6c06d330 con 0x7f4d58003a00 2023-10-14T11:57:57.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.373+0000 7f4d7094c700 1 -- 172.21.15.78:0/473616330 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6c1558c0 con 0x7f4d58004850 2023-10-14T11:57:57.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.373+0000 7f4d7094c700 1 -- 172.21.15.78:0/473616330 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5800bac0 con 0x7f4d6c13dbe0 2023-10-14T11:57:57.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.373+0000 7f4d7194e700 1 --2- 172.21.15.78:0/473616330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d58003a00 0x7f4d58003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:57.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.373+0000 7f4d7214f700 1 --2- 172.21.15.78:0/473616330 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4d6c13dbe0 0x7f4d58005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:57.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.373+0000 7f4d7114d700 1 --2- 172.21.15.78:0/473616330 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d58004850 0x7f4d58004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:57.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.373+0000 7f4d7194e700 1 -- 172.21.15.78:0/473616330 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4d6c13dbe0 msgr2=0x7f4d58005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:57:57.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.373+0000 7f4d7194e700 1 --2- 172.21.15.78:0/473616330 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4d6c13dbe0 0x7f4d58005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:57.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.373+0000 7f4d7194e700 1 -- 172.21.15.78:0/473616330 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d58004850 msgr2=0x7f4d58004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:57:57.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.373+0000 7f4d7194e700 1 --2- 172.21.15.78:0/473616330 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d58004850 0x7f4d58004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:57.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.373+0000 7f4d7194e700 1 -- 172.21.15.78:0/473616330 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d580035b0 con 0x7f4d58003a00 2023-10-14T11:57:57.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.373+0000 7f4d7194e700 1 --2- 172.21.15.78:0/473616330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d58003a00 0x7f4d58003e20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4d68009860 tx=0x7f4d58005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d72c7af375c99217 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:57:57.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.375+0000 7f4d7094c700 1 -- 172.21.15.78:0/473616330 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d6800c070 con 0x7f4d58003a00 2023-10-14T11:57:57.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.375+0000 7f4d7094c700 1 -- 172.21.15.78:0/473616330 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d68004020 con 0x7f4d58003a00 2023-10-14T11:57:57.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.375+0000 7f4d7094c700 1 -- 172.21.15.78:0/473616330 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d680043d0 con 0x7f4d58003a00 2023-10-14T11:57:57.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.376+0000 7f4d73bb2700 1 -- 172.21.15.78:0/473616330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d58003a00 msgr2=0x7f4d58003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:57.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.376+0000 7f4d73bb2700 1 --2- 172.21.15.78:0/473616330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d58003a00 0x7f4d58003e20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4d68009860 tx=0x7f4d58005880 comp rx=0 tx=0).stop 2023-10-14T11:57:57.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.376+0000 7f4d7214f700 1 -- 172.21.15.78:0/473616330 reap_dead start 2023-10-14T11:57:57.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.376+0000 7f4d73bb2700 1 -- 172.21.15.78:0/473616330 shutdown_connections 2023-10-14T11:57:57.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.376+0000 7f4d73bb2700 1 -- 172.21.15.78:0/473616330 >> 172.21.15.78:0/473616330 conn(0x7f4d6c0c3a50 msgr2=0x7f4d6c0c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:57:57.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.376+0000 7f4d73bb2700 1 -- 172.21.15.78:0/473616330 shutdown_connections 2023-10-14T11:57:57.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.376+0000 7f4d73bb2700 1 -- 172.21.15.78:0/473616330 wait complete. 2023-10-14T11:57:57.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.377+0000 7f4d73bb2700 1 Processor -- start 2023-10-14T11:57:57.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.377+0000 7f4d73bb2700 1 -- start start 2023-10-14T11:57:57.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.378+0000 7f4d73bb2700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4d58003a00 0x7f4d6c158ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:57.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.378+0000 7f4d73bb2700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d6c0a8490 0x7f4d6c1f2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:57.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.378+0000 7f4d73bb2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d6c153a30 0x7f4d6c0a93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:57.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.378+0000 7f4d73bb2700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6c0a8910 con 0x7f4d6c153a30 2023-10-14T11:57:57.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.378+0000 7f4d73bb2700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6c1f1800 con 0x7f4d6c0a8490 2023-10-14T11:57:57.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.378+0000 7f4d73bb2700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6c1f2790 con 0x7f4d58003a00 2023-10-14T11:57:57.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.378+0000 7f4d7214f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d6c153a30 0x7f4d6c0a93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:57.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.378+0000 7f4d7194e700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4d58003a00 0x7f4d6c158ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:57.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.378+0000 7f4d7214f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d6c153a30 0x7f4d6c0a93c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43854/0 (socket says 172.21.15.78:43854) 2023-10-14T11:57:57.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.378+0000 7f4d7194e700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4d58003a00 0x7f4d6c158ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.78:43520/0 (socket says 172.21.15.78:43520) 2023-10-14T11:57:57.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.378+0000 7f4d7214f700 1 -- 172.21.15.78:0/2298722250 learned_addr learned my addr 172.21.15.78:0/2298722250 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:57:57.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.378+0000 7f4d7114d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d6c0a8490 0x7f4d6c1f2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:57.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.378+0000 7f4d7214f700 1 -- 172.21.15.78:0/2298722250 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4d58003a00 msgr2=0x7f4d6c158ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:57.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.378+0000 7f4d7214f700 1 --2- 172.21.15.78:0/2298722250 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4d58003a00 0x7f4d6c158ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:57.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.378+0000 7f4d7214f700 1 -- 172.21.15.78:0/2298722250 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d6c0a8490 msgr2=0x7f4d6c1f2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:57.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.378+0000 7f4d7214f700 1 --2- 172.21.15.78:0/2298722250 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d6c0a8490 0x7f4d6c1f2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:57.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.378+0000 7f4d7214f700 1 -- 172.21.15.78:0/2298722250 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d6c157f30 con 0x7f4d6c153a30 2023-10-14T11:57:57.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.379+0000 7f4d7214f700 1 --2- 172.21.15.78:0/2298722250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d6c153a30 0x7f4d6c0a93c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c0712a0 tx=0x7f4d64007db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:57:57.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.380+0000 7f4d627fc700 1 -- 172.21.15.78:0/2298722250 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d6400a440 con 0x7f4d6c153a30 2023-10-14T11:57:57.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.380+0000 7f4d73bb2700 1 -- 172.21.15.78:0/2298722250 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d6c1eed70 con 0x7f4d6c153a30 2023-10-14T11:57:57.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.380+0000 7f4d73bb2700 1 -- 172.21.15.78:0/2298722250 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d6c1f1a50 con 0x7f4d6c153a30 2023-10-14T11:57:57.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.380+0000 7f4d627fc700 1 -- 172.21.15.78:0/2298722250 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d640071f0 con 0x7f4d6c153a30 2023-10-14T11:57:57.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.380+0000 7f4d627fc700 1 -- 172.21.15.78:0/2298722250 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d64003a20 con 0x7f4d6c153a30 2023-10-14T11:57:57.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.380+0000 7f4d627fc700 1 -- 172.21.15.78:0/2298722250 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f4d64003c60 con 0x7f4d6c153a30 2023-10-14T11:57:57.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.380+0000 7f4d627fc700 1 -- 172.21.15.78:0/2298722250 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f4d640041e0 con 0x7f4d6c153a30 2023-10-14T11:57:57.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.381+0000 7f4d73bb2700 1 -- 172.21.15.78:0/2298722250 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d540052f0 con 0x7f4d6c153a30 2023-10-14T11:57:57.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.384+0000 7f4d627fc700 1 -- 172.21.15.78:0/2298722250 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f4d6401b070 con 0x7f4d6c153a30 2023-10-14T11:57:57.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.459+0000 7f4d73bb2700 1 -- 172.21.15.78:0/2298722250 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f4d54005870 con 0x7f4d6c153a30 2023-10-14T11:57:57.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.460+0000 7f4d627fc700 1 -- 172.21.15.78:0/2298722250 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f4d640043f0 con 0x7f4d6c153a30 2023-10-14T11:57:57.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.461+0000 7f4d73bb2700 1 -- 172.21.15.78:0/2298722250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d6c153a30 msgr2=0x7f4d6c0a93c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:57.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.461+0000 7f4d73bb2700 1 --2- 172.21.15.78:0/2298722250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d6c153a30 0x7f4d6c0a93c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c0712a0 tx=0x7f4d64007db0 comp rx=0 tx=0).stop 2023-10-14T11:57:57.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.461+0000 7f4d73bb2700 1 -- 172.21.15.78:0/2298722250 shutdown_connections 2023-10-14T11:57:57.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.461+0000 7f4d73bb2700 1 --2- 172.21.15.78:0/2298722250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d6c153a30 0x7f4d6c0a93c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:57.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.461+0000 7f4d73bb2700 1 --2- 172.21.15.78:0/2298722250 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4d58003a00 0x7f4d6c158ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:57.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.461+0000 7f4d73bb2700 1 --2- 172.21.15.78:0/2298722250 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d6c0a8490 0x7f4d6c1f2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:57.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.461+0000 7f4d73bb2700 1 -- 172.21.15.78:0/2298722250 >> 172.21.15.78:0/2298722250 conn(0x7f4d6c0c3a50 msgr2=0x7f4d6c0c32e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:57:57.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.461+0000 7f4d73bb2700 1 -- 172.21.15.78:0/2298722250 shutdown_connections 2023-10-14T11:57:57.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:57:57.461+0000 7f4d73bb2700 1 -- 172.21.15.78:0/2298722250 wait complete. 2023-10-14T11:57:57.465 INFO:teuthology.orchestra.run.smithi078.stderr:adjusted tunables profile to default 2023-10-14T11:57:57.474 INFO:tasks.ceph:check_enable_crimson: False 2023-10-14T11:57:57.474 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-14T11:57:57.475 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-14T11:57:57.475 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-14T11:57:57.488 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-10-14T11:57:57.487+0000 7f94eb2a9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T11:57:57.496 INFO:tasks.ceph.mgr.y.smithi078.stderr:2023-10-14T11:57:57.495+0000 7ff4d96d6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T11:57:57.501 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-14T11:57:57.501 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-14T11:57:57.553 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-10-14T11:57:57.552+0000 7f94eb2a9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T11:57:57.564 INFO:tasks.ceph.mgr.y.smithi078.stderr:2023-10-14T11:57:57.563+0000 7ff4d96d6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T11:57:57.567 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-14T11:57:57.567 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-14T11:57:57.628 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-10-14T11:57:57.627+0000 7f94eb2a9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T11:57:57.634 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-14T11:57:57.634 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-10-14T11:57:57.644 INFO:tasks.ceph.mgr.y.smithi078.stderr:2023-10-14T11:57:57.643+0000 7ff4d96d6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T11:57:57.669 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-14T11:57:57.670 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-10-14T11:57:57.696 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-14T11:57:57.696 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-10-14T11:57:57.708 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-10-14T11:57:57.706+0000 7f94eb2a9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T11:57:57.726 INFO:tasks.ceph.mgr.y.smithi078.stderr:2023-10-14T11:57:57.726+0000 7ff4d96d6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T11:57:57.733 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-14T11:57:57.733 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-10-14T11:57:57.799 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-14T11:57:57.800 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-10-14T11:57:57.857 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-10-14T11:57:57.855+0000 7f94eb2a9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T11:57:57.865 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 625a5fc0-9474-4eae-b80e-14c27a74ad19 0 2023-10-14T11:57:57.867 INFO:tasks.ceph.mgr.y.smithi078.stderr:2023-10-14T11:57:57.866+0000 7ff4d96d6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T11:57:58.026 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.024+0000 7f515a99e700 1 Processor -- start 2023-10-14T11:57:58.027 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.025+0000 7f515a99e700 1 -- start start 2023-10-14T11:57:58.029 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.025+0000 7f515a99e700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f515414c850 0x7f51540c70b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:58.029 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.025+0000 7f515a99e700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f515406e4d0 0x7f515406e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:58.030 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.025+0000 7f515a99e700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f515406e040 con 0x7f51540bd110 2023-10-14T11:57:58.030 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.025+0000 7f515a99e700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f515406d7e0 con 0x7f515406e4d0 2023-10-14T11:57:58.030 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.025+0000 7f515a99e700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5154153390 con 0x7f515414c850 2023-10-14T11:57:58.030 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.026+0000 7f515999c700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f515414c850 0x7f51540c70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:58.031 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.026+0000 7f515999c700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f515414c850 0x7f51540c70b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.195:39428/0 (socket says 172.21.15.195:39428) 2023-10-14T11:57:58.031 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.026+0000 7f515999c700 1 -- 172.21.15.195:0/1603089799 learned_addr learned my addr 172.21.15.195:0/1603089799 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-14T11:57:58.031 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.026+0000 7f515a19d700 1 --2- 172.21.15.195:0/1603089799 >> v2:172.21.15.78:3300/0 conn(0x7f515406e4d0 0x7f515406e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:58.031 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.026+0000 7f515999c700 1 -- 172.21.15.195:0/1603089799 >> v2:172.21.15.78:3300/0 conn(0x7f515406e4d0 msgr2=0x7f515406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:57:58.031 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.026+0000 7f515999c700 1 --2- 172.21.15.195:0/1603089799 >> v2:172.21.15.78:3300/0 conn(0x7f515406e4d0 0x7f515406e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.032 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.026+0000 7f515999c700 1 -- 172.21.15.195:0/1603089799 >> v1:172.21.15.78:6789/0 conn(0x7f51540bd110 legacy=0x7f51540bc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:58.032 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.026+0000 7f515999c700 1 -- 172.21.15.195:0/1603089799 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f515406d9f0 con 0x7f515414c850 2023-10-14T11:57:58.032 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.026+0000 7f515999c700 1 --2- 172.21.15.195:0/1603089799 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f515414c850 0x7f51540c70b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f51500098f0 tx=0x7f5150009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b10523d6ccdfe29 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:57:58.032 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.026+0000 7f515899a700 1 -- 172.21.15.195:0/1603089799 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5150011830 con 0x7f515414c850 2023-10-14T11:57:58.032 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.026+0000 7f515899a700 1 -- 172.21.15.195:0/1603089799 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5150011990 con 0x7f515414c850 2023-10-14T11:57:58.033 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.026+0000 7f515899a700 1 -- 172.21.15.195:0/1603089799 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5150011d20 con 0x7f515414c850 2023-10-14T11:57:58.033 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.027+0000 7f515a99e700 1 -- 172.21.15.195:0/1603089799 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f515414c850 msgr2=0x7f51540c70b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:58.033 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.027+0000 7f515a99e700 1 --2- 172.21.15.195:0/1603089799 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f515414c850 0x7f51540c70b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f51500098f0 tx=0x7f5150009c00 comp rx=0 tx=0).stop 2023-10-14T11:57:58.033 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.027+0000 7f515a99e700 1 -- 172.21.15.195:0/1603089799 shutdown_connections 2023-10-14T11:57:58.033 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.027+0000 7f515a99e700 1 --2- 172.21.15.195:0/1603089799 >> v2:172.21.15.78:3300/0 conn(0x7f515406e4d0 0x7f515406e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.034 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.027+0000 7f515a99e700 1 --2- 172.21.15.195:0/1603089799 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f515414c850 0x7f51540c70b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.034 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.027+0000 7f515a99e700 1 -- 172.21.15.195:0/1603089799 >> 172.21.15.195:0/1603089799 conn(0x7f51540c6590 msgr2=0x7f51540c69a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:57:58.034 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.027+0000 7f515a99e700 1 -- 172.21.15.195:0/1603089799 shutdown_connections 2023-10-14T11:57:58.034 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.027+0000 7f515a99e700 1 -- 172.21.15.195:0/1603089799 wait complete. 2023-10-14T11:57:58.036 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.028+0000 7f515a99e700 1 Processor -- start 2023-10-14T11:57:58.037 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.028+0000 7f515a99e700 1 -- start start 2023-10-14T11:57:58.037 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.029+0000 7f515a99e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f515406e4d0 0x7f51541e7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:58.037 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.029+0000 7f515a99e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51540bd110 0x7f515414e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:58.037 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.029+0000 7f515a99e700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f51540bde50 0x7f5154155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:58.037 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.029+0000 7f515a99e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515414e860 con 0x7f51540bd110 2023-10-14T11:57:58.038 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.029+0000 7f515a99e700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5154154c60 con 0x7f515406e4d0 2023-10-14T11:57:58.038 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.029+0000 7f515a99e700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5154154f00 con 0x7f51540bde50 2023-10-14T11:57:58.038 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.029+0000 7f515a19d700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f51540bde50 0x7f5154155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:58.038 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.029+0000 7f515a19d700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f51540bde50 0x7f5154155cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.195:39438/0 (socket says 172.21.15.195:39438) 2023-10-14T11:57:58.038 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.029+0000 7f515a19d700 1 -- 172.21.15.195:0/3751409610 learned_addr learned my addr 172.21.15.195:0/3751409610 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-14T11:57:58.039 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.029+0000 7f515999c700 1 --2- 172.21.15.195:0/3751409610 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f515406e4d0 0x7f51541e7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:58.039 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.029+0000 7f515919b700 1 --2- 172.21.15.195:0/3751409610 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51540bd110 0x7f515414e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:58.039 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.029+0000 7f515a19d700 1 -- 172.21.15.195:0/3751409610 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f515406e4d0 msgr2=0x7f51541e7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:58.039 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.029+0000 7f515a19d700 1 --2- 172.21.15.195:0/3751409610 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f515406e4d0 0x7f51541e7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.039 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.029+0000 7f515a19d700 1 -- 172.21.15.195:0/3751409610 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51540bd110 msgr2=0x7f515414e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:58.040 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.029+0000 7f515a19d700 1 --2- 172.21.15.195:0/3751409610 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51540bd110 0x7f515414e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.040 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.029+0000 7f515a19d700 1 -- 172.21.15.195:0/3751409610 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5150009550 con 0x7f51540bde50 2023-10-14T11:57:58.040 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.029+0000 7f515a19d700 1 --2- 172.21.15.195:0/3751409610 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f51540bde50 0x7f5154155cc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f514c00efc0 tx=0x7f514c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:57:58.040 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.030+0000 7f514a7fc700 1 -- 172.21.15.195:0/3751409610 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514c009ea0 con 0x7f51540bde50 2023-10-14T11:57:58.040 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.030+0000 7f514a7fc700 1 -- 172.21.15.195:0/3751409610 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f514c00f040 con 0x7f51540bde50 2023-10-14T11:57:58.041 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.030+0000 7f515a99e700 1 -- 172.21.15.195:0/3751409610 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51541551e0 con 0x7f51540bde50 2023-10-14T11:57:58.041 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.030+0000 7f514a7fc700 1 -- 172.21.15.195:0/3751409610 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514c00ced0 con 0x7f51540bde50 2023-10-14T11:57:58.041 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.030+0000 7f515a99e700 1 -- 172.21.15.195:0/3751409610 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51541557a0 con 0x7f51540bde50 2023-10-14T11:57:58.041 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.030+0000 7f514a7fc700 1 -- 172.21.15.195:0/3751409610 <== mon.2 v2:172.21.15.195:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f514c014780 con 0x7f51540bde50 2023-10-14T11:57:58.041 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.030+0000 7f514a7fc700 1 -- 172.21.15.195:0/3751409610 <== mon.2 v2:172.21.15.195:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f514c01d9e0 con 0x7f51540bde50 2023-10-14T11:57:58.041 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.031+0000 7f515a99e700 1 -- 172.21.15.195:0/3751409610 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51541559c0 con 0x7f51540bde50 2023-10-14T11:57:58.042 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.033+0000 7f514a7fc700 1 -- 172.21.15.195:0/3751409610 <== mon.2 v2:172.21.15.195:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f514c03bb40 con 0x7f51540bde50 2023-10-14T11:57:58.118 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.116+0000 7f515a99e700 1 -- 172.21.15.195:0/3751409610 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_command({"prefix": "osd new", "uuid": "625a5fc0-9474-4eae-b80e-14c27a74ad19", "id": 0} v 0) v1 -- 0x7f515404f400 con 0x7f51540bde50 2023-10-14T11:57:58.121 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.119+0000 7f514a7fc700 1 -- 172.21.15.195:0/3751409610 <== mon.2 v2:172.21.15.195:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "625a5fc0-9474-4eae-b80e-14c27a74ad19", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f514c02a070 con 0x7f51540bde50 2023-10-14T11:57:58.121 INFO:teuthology.orchestra.run.smithi195.stdout:0 2023-10-14T11:57:58.123 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.120+0000 7f515a99e700 1 -- 172.21.15.195:0/3751409610 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f51540bde50 msgr2=0x7f5154155cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:58.123 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.120+0000 7f515a99e700 1 --2- 172.21.15.195:0/3751409610 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f51540bde50 0x7f5154155cc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f514c00efc0 tx=0x7f514c00c3a0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.123 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.120+0000 7f515a99e700 1 -- 172.21.15.195:0/3751409610 shutdown_connections 2023-10-14T11:57:58.123 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.120+0000 7f515a99e700 1 --2- 172.21.15.195:0/3751409610 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f51540bde50 0x7f5154155cc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.123 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.120+0000 7f515a99e700 1 --2- 172.21.15.195:0/3751409610 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f515406e4d0 0x7f51541e7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.124 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.120+0000 7f515a99e700 1 --2- 172.21.15.195:0/3751409610 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51540bd110 0x7f515414e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.124 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.120+0000 7f515a99e700 1 -- 172.21.15.195:0/3751409610 >> 172.21.15.195:0/3751409610 conn(0x7f51540c6590 msgr2=0x7f51540c4ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:57:58.124 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.120+0000 7f515a99e700 1 -- 172.21.15.195:0/3751409610 shutdown_connections 2023-10-14T11:57:58.124 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.121+0000 7f515a99e700 1 -- 172.21.15.195:0/3751409610 wait complete. 2023-10-14T11:57:58.134 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 49d63db1-2981-49ab-9c40-c50fc744b982 1 2023-10-14T11:57:58.255 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.253+0000 7f8609540700 1 Processor -- start 2023-10-14T11:57:58.255 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.253+0000 7f8609540700 1 -- start start 2023-10-14T11:57:58.258 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.254+0000 7f8609540700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f86040c3490 0x7f8604156690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:58.258 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.254+0000 7f8609540700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86040c3c00 con 0x7f8604072640 2023-10-14T11:57:58.258 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.254+0000 7f8609540700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8604156bd0 con 0x7f860406ab40 2023-10-14T11:57:58.258 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.254+0000 7f8609540700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86040c0bc0 con 0x7f86040c3490 2023-10-14T11:57:58.259 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.254+0000 7f86037fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f86040c3490 0x7f8604156690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:58.259 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.254+0000 7f8602ffd700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f860406ab40 0x7f860414fee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:39586/0 (socket says 172.21.15.195:39586) 2023-10-14T11:57:58.259 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.254+0000 7f8602ffd700 1 -- 172.21.15.195:0/4026632458 learned_addr learned my addr 172.21.15.195:0/4026632458 (peer_addr_for_me v1:172.21.15.195:0/0) 2023-10-14T11:57:58.259 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.254+0000 7f8601ffb700 1 -- 172.21.15.195:0/4026632458 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 563927994 0 0) 0x7f8604156bd0 con 0x7f860406ab40 2023-10-14T11:57:58.259 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.254+0000 7f8601ffb700 1 -- 172.21.15.195:0/4026632458 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85ec0035b0 con 0x7f860406ab40 2023-10-14T11:57:58.260 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.254+0000 7f86037fe700 1 -- 172.21.15.195:0/4026632458 >> v1:172.21.15.195:6789/0 conn(0x7f860406ab40 legacy=0x7f860414fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:58.260 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.254+0000 7f86037fe700 1 -- 172.21.15.195:0/4026632458 >> v1:172.21.15.78:6789/0 conn(0x7f8604072640 legacy=0x7f8604155f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:58.260 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.254+0000 7f86037fe700 1 -- 172.21.15.195:0/4026632458 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86040c25e0 con 0x7f86040c3490 2023-10-14T11:57:58.260 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.255+0000 7f86037fe700 1 --2- 172.21.15.195:0/4026632458 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86040c3490 0x7f8604156690 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f85fc009d90 tx=0x7f85fc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92d86201acb57b1a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:57:58.260 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.255+0000 7f8601ffb700 1 -- 172.21.15.195:0/4026632458 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85fc00e070 con 0x7f86040c3490 2023-10-14T11:57:58.261 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.255+0000 7f8601ffb700 1 -- 172.21.15.195:0/4026632458 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85fc00bb20 con 0x7f86040c3490 2023-10-14T11:57:58.261 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.255+0000 7f8601ffb700 1 -- 172.21.15.195:0/4026632458 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85fc012430 con 0x7f86040c3490 2023-10-14T11:57:58.261 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.255+0000 7f8609540700 1 -- 172.21.15.195:0/4026632458 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86040c3490 msgr2=0x7f8604156690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:58.261 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.255+0000 7f8609540700 1 --2- 172.21.15.195:0/4026632458 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86040c3490 0x7f8604156690 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f85fc009d90 tx=0x7f85fc00b330 comp rx=0 tx=0).stop 2023-10-14T11:57:58.261 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.255+0000 7f8609540700 1 -- 172.21.15.195:0/4026632458 shutdown_connections 2023-10-14T11:57:58.262 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.255+0000 7f8609540700 1 --2- 172.21.15.195:0/4026632458 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86040c3490 0x7f8604156690 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.262 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.255+0000 7f8609540700 1 -- 172.21.15.195:0/4026632458 >> 172.21.15.195:0/4026632458 conn(0x7f86040a84a0 msgr2=0x7f86040a88b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:57:58.262 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.255+0000 7f8609540700 1 -- 172.21.15.195:0/4026632458 shutdown_connections 2023-10-14T11:57:58.262 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.255+0000 7f8609540700 1 -- 172.21.15.195:0/4026632458 wait complete. 2023-10-14T11:57:58.265 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.257+0000 7f8609540700 1 Processor -- start 2023-10-14T11:57:58.265 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.257+0000 7f8609540700 1 -- start start 2023-10-14T11:57:58.265 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.257+0000 7f8609540700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f860406ab40 0x7f86040bf280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:58.266 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.257+0000 7f8609540700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8604072640 0x7f86041ed340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:58.266 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.257+0000 7f8609540700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86040c3490 0x7f86040be360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:58.267 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.257+0000 7f8609540700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86040c4610 con 0x7f86040c3490 2023-10-14T11:57:58.267 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.257+0000 7f8609540700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86041556f0 con 0x7f860406ab40 2023-10-14T11:57:58.267 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.257+0000 7f8609540700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86040c4e70 con 0x7f8604072640 2023-10-14T11:57:58.268 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.258+0000 7f86027fc700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8604072640 0x7f86041ed340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:58.268 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.258+0000 7f86027fc700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8604072640 0x7f86041ed340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.195:39448/0 (socket says 172.21.15.195:39448) 2023-10-14T11:57:58.268 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.258+0000 7f86027fc700 1 -- 172.21.15.195:0/3218715810 learned_addr learned my addr 172.21.15.195:0/3218715810 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-14T11:57:58.268 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.258+0000 7f86037fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86040c3490 0x7f86040be360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:58.268 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.258+0000 7f8602ffd700 1 --2- 172.21.15.195:0/3218715810 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f860406ab40 0x7f86040bf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:58.269 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.258+0000 7f8602ffd700 1 -- 172.21.15.195:0/3218715810 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8604072640 msgr2=0x7f86041ed340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:58.269 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.258+0000 7f8602ffd700 1 --2- 172.21.15.195:0/3218715810 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8604072640 0x7f86041ed340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.269 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.258+0000 7f8602ffd700 1 -- 172.21.15.195:0/3218715810 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86040c3490 msgr2=0x7f86040be360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:58.269 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.258+0000 7f8602ffd700 1 --2- 172.21.15.195:0/3218715810 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86040c3490 0x7f86040be360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.269 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.258+0000 7f8602ffd700 1 -- 172.21.15.195:0/3218715810 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85fc009a70 con 0x7f860406ab40 2023-10-14T11:57:58.270 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.258+0000 7f86037fe700 1 --2- 172.21.15.195:0/3218715810 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86040c3490 0x7f86040be360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T11:57:58.270 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.258+0000 7f86027fc700 1 --2- 172.21.15.195:0/3218715810 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8604072640 0x7f86041ed340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T11:57:58.270 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.258+0000 7f8602ffd700 1 --2- 172.21.15.195:0/3218715810 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f860406ab40 0x7f86040bf280 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f85f400c490 tx=0x7f85f400c7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:57:58.270 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.258+0000 7f85eb7fe700 1 -- 172.21.15.195:0/3218715810 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f400cea0 con 0x7f860406ab40 2023-10-14T11:57:58.270 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.258+0000 7f85eb7fe700 1 -- 172.21.15.195:0/3218715810 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f4005d70 con 0x7f860406ab40 2023-10-14T11:57:58.271 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.258+0000 7f8609540700 1 -- 172.21.15.195:0/3218715810 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86041edc10 con 0x7f860406ab40 2023-10-14T11:57:58.271 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.258+0000 7f85eb7fe700 1 -- 172.21.15.195:0/3218715810 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f40072f0 con 0x7f860406ab40 2023-10-14T11:57:58.271 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.258+0000 7f8609540700 1 -- 172.21.15.195:0/3218715810 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8604155350 con 0x7f860406ab40 2023-10-14T11:57:58.271 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.258+0000 7f85eb7fe700 1 -- 172.21.15.195:0/3218715810 <== mon.1 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f85f401c440 con 0x7f860406ab40 2023-10-14T11:57:58.271 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.258+0000 7f85eb7fe700 1 -- 172.21.15.195:0/3218715810 <== mon.1 v2:172.21.15.195:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f85f40127b0 con 0x7f860406ab40 2023-10-14T11:57:58.271 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.259+0000 7f8609540700 1 -- 172.21.15.195:0/3218715810 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86041eda60 con 0x7f860406ab40 2023-10-14T11:57:58.272 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.261+0000 7f85eb7fe700 1 -- 172.21.15.195:0/3218715810 <== mon.1 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f85f4017070 con 0x7f860406ab40 2023-10-14T11:57:58.272 INFO:tasks.ceph.mgr.y.smithi078.stderr:2023-10-14T11:57:58.263+0000 7ff4d96d6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T11:57:58.273 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-10-14T11:57:58.267+0000 7f94eb2a9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T11:57:58.346 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.345+0000 7f8609540700 1 -- 172.21.15.195:0/3218715810 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd new", "uuid": "49d63db1-2981-49ab-9c40-c50fc744b982", "id": 1} v 0) v1 -- 0x7f86041edf60 con 0x7f860406ab40 2023-10-14T11:57:58.349 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.348+0000 7f85eb7fe700 1 -- 172.21.15.195:0/3218715810 <== mon.1 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "49d63db1-2981-49ab-9c40-c50fc744b982", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f85f4012970 con 0x7f860406ab40 2023-10-14T11:57:58.350 INFO:teuthology.orchestra.run.smithi195.stdout:1 2023-10-14T11:57:58.351 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.349+0000 7f8609540700 1 -- 172.21.15.195:0/3218715810 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f860406ab40 msgr2=0x7f86040bf280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:58.351 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.349+0000 7f8609540700 1 --2- 172.21.15.195:0/3218715810 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f860406ab40 0x7f86040bf280 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f85f400c490 tx=0x7f85f400c7a0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.351 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.349+0000 7f8609540700 1 -- 172.21.15.195:0/3218715810 shutdown_connections 2023-10-14T11:57:58.351 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.349+0000 7f8609540700 1 --2- 172.21.15.195:0/3218715810 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86040c3490 0x7f86040be360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.352 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.349+0000 7f8609540700 1 --2- 172.21.15.195:0/3218715810 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f860406ab40 0x7f86040bf280 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.352 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.349+0000 7f8609540700 1 --2- 172.21.15.195:0/3218715810 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8604072640 0x7f86041ed340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.352 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.349+0000 7f8609540700 1 -- 172.21.15.195:0/3218715810 >> 172.21.15.195:0/3218715810 conn(0x7f86040a84a0 msgr2=0x7f86041565f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:57:58.352 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.349+0000 7f8609540700 1 -- 172.21.15.195:0/3218715810 shutdown_connections 2023-10-14T11:57:58.352 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.349+0000 7f8609540700 1 -- 172.21.15.195:0/3218715810 wait complete. 2023-10-14T11:57:58.364 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 1d7d0b29-cbd1-4717-9fe6-765a8d4998b3 2 2023-10-14T11:57:58.381 INFO:tasks.ceph.mgr.y.smithi078.stderr:2023-10-14T11:57:58.380+0000 7ff4d96d6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T11:57:58.393 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-10-14T11:57:58.391+0000 7f94eb2a9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T11:57:58.464 INFO:tasks.ceph.mgr.y.smithi078.stderr:2023-10-14T11:57:58.463+0000 7ff4d96d6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T11:57:58.480 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-10-14T11:57:58.478+0000 7f94eb2a9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T11:57:58.485 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.483+0000 7ff4898fc700 1 Processor -- start 2023-10-14T11:57:58.485 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.483+0000 7ff4898fc700 1 -- start start 2023-10-14T11:57:58.487 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.484+0000 7ff4898fc700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff48413dbe0 0x7ff48406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:58.488 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.484+0000 7ff4898fc700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff4840c4540 0x7ff4840c02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:58.488 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.484+0000 7ff4898fc700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff48406ebe0 con 0x7ff48406ae10 2023-10-14T11:57:58.488 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.484+0000 7ff4898fc700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff48406d330 con 0x7ff48413dbe0 2023-10-14T11:57:58.488 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.484+0000 7ff4898fc700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4841558c0 con 0x7ff4840c4540 2023-10-14T11:57:58.488 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.484+0000 7ff4827fc700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff4840c4540 0x7ff4840c02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:58.489 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.484+0000 7ff4827fc700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff4840c4540 0x7ff4840c02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44870/0 (socket says 172.21.15.195:44870) 2023-10-14T11:57:58.489 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.484+0000 7ff4827fc700 1 -- 172.21.15.195:0/3034077540 learned_addr learned my addr 172.21.15.195:0/3034077540 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-14T11:57:58.489 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.484+0000 7ff482ffd700 1 --2- 172.21.15.195:0/3034077540 >> v2:172.21.15.78:3300/0 conn(0x7ff48413dbe0 0x7ff48406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:58.489 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.484+0000 7ff4827fc700 1 -- 172.21.15.195:0/3034077540 >> v2:172.21.15.78:3300/0 conn(0x7ff48413dbe0 msgr2=0x7ff48406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:57:58.489 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.484+0000 7ff4827fc700 1 --2- 172.21.15.195:0/3034077540 >> v2:172.21.15.78:3300/0 conn(0x7ff48413dbe0 0x7ff48406a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.490 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.484+0000 7ff4827fc700 1 -- 172.21.15.195:0/3034077540 >> v1:172.21.15.78:6789/0 conn(0x7ff48406ae10 legacy=0x7ff4840c0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:58.490 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.484+0000 7ff4827fc700 1 -- 172.21.15.195:0/3034077540 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff484154b00 con 0x7ff4840c4540 2023-10-14T11:57:58.490 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.485+0000 7ff4827fc700 1 --2- 172.21.15.195:0/3034077540 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff4840c4540 0x7ff4840c02e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff48413e360 tx=0x7ff478009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f93d4d906a6d2f10 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:57:58.490 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.485+0000 7ff481ffb700 1 -- 172.21.15.195:0/3034077540 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff478011830 con 0x7ff4840c4540 2023-10-14T11:57:58.490 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.485+0000 7ff481ffb700 1 -- 172.21.15.195:0/3034077540 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff478011990 con 0x7ff4840c4540 2023-10-14T11:57:58.491 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.485+0000 7ff481ffb700 1 -- 172.21.15.195:0/3034077540 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff478011d20 con 0x7ff4840c4540 2023-10-14T11:57:58.491 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.485+0000 7ff4898fc700 1 -- 172.21.15.195:0/3034077540 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff4840c4540 msgr2=0x7ff4840c02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:58.491 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.485+0000 7ff4898fc700 1 --2- 172.21.15.195:0/3034077540 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff4840c4540 0x7ff4840c02e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff48413e360 tx=0x7ff478009bd0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.491 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.485+0000 7ff4898fc700 1 -- 172.21.15.195:0/3034077540 shutdown_connections 2023-10-14T11:57:58.492 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.485+0000 7ff4898fc700 1 --2- 172.21.15.195:0/3034077540 >> v2:172.21.15.78:3300/0 conn(0x7ff48413dbe0 0x7ff48406a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.492 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.485+0000 7ff4898fc700 1 --2- 172.21.15.195:0/3034077540 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff4840c4540 0x7ff4840c02e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.492 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.485+0000 7ff4898fc700 1 -- 172.21.15.195:0/3034077540 >> 172.21.15.195:0/3034077540 conn(0x7ff4840c3a50 msgr2=0x7ff4840c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:57:58.492 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.485+0000 7ff4898fc700 1 -- 172.21.15.195:0/3034077540 shutdown_connections 2023-10-14T11:57:58.492 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.486+0000 7ff4898fc700 1 -- 172.21.15.195:0/3034077540 wait complete. 2023-10-14T11:57:58.495 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.487+0000 7ff4898fc700 1 Processor -- start 2023-10-14T11:57:58.495 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.487+0000 7ff4898fc700 1 -- start start 2023-10-14T11:57:58.495 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.487+0000 7ff4898fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff48406ae10 0x7ff484137990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:58.495 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.487+0000 7ff4898fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff4840c4540 0x7ff484137ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:58.496 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.487+0000 7ff4898fc700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff48413dbe0 0x7ff484135300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:58.496 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.487+0000 7ff4898fc700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff484151dc0 con 0x7ff48406ae10 2023-10-14T11:57:58.496 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.487+0000 7ff4898fc700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4841ea490 con 0x7ff4840c4540 2023-10-14T11:57:58.496 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.487+0000 7ff4898fc700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff484151770 con 0x7ff48413dbe0 2023-10-14T11:57:58.497 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.488+0000 7ff4827fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff4840c4540 0x7ff484137ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:58.497 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.488+0000 7ff4837fe700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff48413dbe0 0x7ff484135300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:58.497 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.488+0000 7ff4827fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff4840c4540 0x7ff484137ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44876/0 (socket says 172.21.15.195:44876) 2023-10-14T11:57:58.497 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.488+0000 7ff4827fc700 1 -- 172.21.15.195:0/1475114061 learned_addr learned my addr 172.21.15.195:0/1475114061 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-14T11:57:58.497 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.488+0000 7ff482ffd700 1 --2- 172.21.15.195:0/1475114061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff48406ae10 0x7ff484137990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:58.498 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.488+0000 7ff4837fe700 1 -- 172.21.15.195:0/1475114061 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff4840c4540 msgr2=0x7ff484137ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:58.498 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.488+0000 7ff4837fe700 1 --2- 172.21.15.195:0/1475114061 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff4840c4540 0x7ff484137ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.498 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.488+0000 7ff4837fe700 1 -- 172.21.15.195:0/1475114061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff48406ae10 msgr2=0x7ff484137990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:58.498 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.488+0000 7ff4837fe700 1 --2- 172.21.15.195:0/1475114061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff48406ae10 0x7ff484137990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.498 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.488+0000 7ff4837fe700 1 -- 172.21.15.195:0/1475114061 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff478009550 con 0x7ff48413dbe0 2023-10-14T11:57:58.499 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.488+0000 7ff4827fc700 1 --2- 172.21.15.195:0/1475114061 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff4840c4540 0x7ff484137ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T11:57:58.499 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.488+0000 7ff482ffd700 1 --2- 172.21.15.195:0/1475114061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff48406ae10 0x7ff484137990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T11:57:58.499 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.488+0000 7ff4837fe700 1 --2- 172.21.15.195:0/1475114061 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff48413dbe0 0x7ff484135300 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff47c00cae0 tx=0x7ff47c00cdf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:57:58.499 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.488+0000 7ff46bfff700 1 -- 172.21.15.195:0/1475114061 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47c003c00 con 0x7ff48413dbe0 2023-10-14T11:57:58.499 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.488+0000 7ff46bfff700 1 -- 172.21.15.195:0/1475114061 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff47c003d60 con 0x7ff48413dbe0 2023-10-14T11:57:58.500 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.488+0000 7ff4898fc700 1 -- 172.21.15.195:0/1475114061 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff484151a50 con 0x7ff48413dbe0 2023-10-14T11:57:58.500 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.488+0000 7ff46bfff700 1 -- 172.21.15.195:0/1475114061 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47c00a980 con 0x7ff48413dbe0 2023-10-14T11:57:58.500 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.488+0000 7ff4898fc700 1 -- 172.21.15.195:0/1475114061 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4840c6b60 con 0x7ff48413dbe0 2023-10-14T11:57:58.500 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.488+0000 7ff46bfff700 1 -- 172.21.15.195:0/1475114061 <== mon.2 v2:172.21.15.195:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff47c0050d0 con 0x7ff48413dbe0 2023-10-14T11:57:58.500 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.488+0000 7ff46bfff700 1 -- 172.21.15.195:0/1475114061 <== mon.2 v2:172.21.15.195:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7ff47c012840 con 0x7ff48413dbe0 2023-10-14T11:57:58.501 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.489+0000 7ff4898fc700 1 -- 172.21.15.195:0/1475114061 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4840c6db0 con 0x7ff48413dbe0 2023-10-14T11:57:58.501 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.491+0000 7ff46bfff700 1 -- 172.21.15.195:0/1475114061 <== mon.2 v2:172.21.15.195:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7ff47c012420 con 0x7ff48413dbe0 2023-10-14T11:57:58.545 INFO:tasks.ceph.mgr.y.smithi078.stderr:2023-10-14T11:57:58.544+0000 7ff4d96d6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T11:57:58.566 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-10-14T11:57:58.564+0000 7f94eb2a9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T11:57:58.577 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.573+0000 7ff4898fc700 1 -- 172.21.15.195:0/1475114061 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_command({"prefix": "osd new", "uuid": "1d7d0b29-cbd1-4717-9fe6-765a8d4998b3", "id": 2} v 0) v1 -- 0x7ff484154d90 con 0x7ff48413dbe0 2023-10-14T11:57:58.580 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.579+0000 7ff46bfff700 1 -- 172.21.15.195:0/1475114061 <== mon.2 v2:172.21.15.195:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1d7d0b29-cbd1-4717-9fe6-765a8d4998b3", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff47c012640 con 0x7ff48413dbe0 2023-10-14T11:57:58.581 INFO:teuthology.orchestra.run.smithi195.stdout:2 2023-10-14T11:57:58.582 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.579+0000 7ff4898fc700 1 -- 172.21.15.195:0/1475114061 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff48413dbe0 msgr2=0x7ff484135300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:58.582 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.580+0000 7ff4898fc700 1 --2- 172.21.15.195:0/1475114061 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff48413dbe0 0x7ff484135300 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff47c00cae0 tx=0x7ff47c00cdf0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.582 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.580+0000 7ff4898fc700 1 -- 172.21.15.195:0/1475114061 shutdown_connections 2023-10-14T11:57:58.582 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.580+0000 7ff4898fc700 1 --2- 172.21.15.195:0/1475114061 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff48413dbe0 0x7ff484135300 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.583 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.580+0000 7ff4898fc700 1 --2- 172.21.15.195:0/1475114061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff48406ae10 0x7ff484137990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.583 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.580+0000 7ff4898fc700 1 --2- 172.21.15.195:0/1475114061 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff4840c4540 0x7ff484137ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.583 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.580+0000 7ff4898fc700 1 -- 172.21.15.195:0/1475114061 >> 172.21.15.195:0/1475114061 conn(0x7ff4840c3a50 msgr2=0x7ff4840c02e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:57:58.583 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.580+0000 7ff4898fc700 1 -- 172.21.15.195:0/1475114061 shutdown_connections 2023-10-14T11:57:58.583 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.580+0000 7ff4898fc700 1 -- 172.21.15.195:0/1475114061 wait complete. 2023-10-14T11:57:58.593 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 2f75ded1-e9e2-40e7-99ae-45e1e425e1f0 3 2023-10-14T11:57:58.757 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.756+0000 7f72671b3700 1 Processor -- start 2023-10-14T11:57:58.758 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.756+0000 7f72671b3700 1 -- start start 2023-10-14T11:57:58.760 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.757+0000 7f72671b3700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f726014ed10 0x7f7260152ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:58.761 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.757+0000 7f72671b3700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7260154c20 0x7f7260155030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:58.761 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.757+0000 7f72671b3700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72600c26d0 con 0x7f72600c35e0 2023-10-14T11:57:58.761 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.757+0000 7f72671b3700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72600a8520 con 0x7f7260154c20 2023-10-14T11:57:58.761 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.757+0000 7f72671b3700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72600c29b0 con 0x7f726014ed10 2023-10-14T11:57:58.761 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.757+0000 7f7264f4f700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f726014ed10 0x7f7260152ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:58.762 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.757+0000 7f725ffff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f72600c35e0 0x7f7260153400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:39610/0 (socket says 172.21.15.195:39610) 2023-10-14T11:57:58.762 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.757+0000 7f7264f4f700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f726014ed10 0x7f7260152ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44880/0 (socket says 172.21.15.195:44880) 2023-10-14T11:57:58.762 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.757+0000 7f725ffff700 1 -- 172.21.15.195:0/1501059861 learned_addr learned my addr 172.21.15.195:0/1501059861 (peer_addr_for_me v1:172.21.15.195:0/0) 2023-10-14T11:57:58.762 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.757+0000 7f7265750700 1 --2- 172.21.15.195:0/1501059861 >> v2:172.21.15.78:3300/0 conn(0x7f7260154c20 0x7f7260155030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:58.762 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.757+0000 7f7264f4f700 1 -- 172.21.15.195:0/1501059861 >> v2:172.21.15.78:3300/0 conn(0x7f7260154c20 msgr2=0x7f7260155030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:57:58.763 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.757+0000 7f7264f4f700 1 --2- 172.21.15.195:0/1501059861 >> v2:172.21.15.78:3300/0 conn(0x7f7260154c20 0x7f7260155030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.763 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.757+0000 7f7264f4f700 1 -- 172.21.15.195:0/1501059861 >> v1:172.21.15.195:6789/0 conn(0x7f72600c35e0 legacy=0x7f7260153400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:58.763 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.757+0000 7f7264f4f700 1 -- 172.21.15.195:0/1501059861 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72600c2c90 con 0x7f726014ed10 2023-10-14T11:57:58.763 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.758+0000 7f7264f4f700 1 --2- 172.21.15.195:0/1501059861 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f726014ed10 0x7f7260152ec0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7260072ad0 tx=0x7f7254009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=888ea74878d8cc13 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:57:58.763 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.758+0000 7f725f7fe700 1 -- 172.21.15.195:0/1501059861 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7254004230 con 0x7f726014ed10 2023-10-14T11:57:58.764 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.758+0000 7f725f7fe700 1 -- 172.21.15.195:0/1501059861 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7254004390 con 0x7f726014ed10 2023-10-14T11:57:58.764 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.758+0000 7f725f7fe700 1 -- 172.21.15.195:0/1501059861 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72540036a0 con 0x7f726014ed10 2023-10-14T11:57:58.764 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.758+0000 7f72671b3700 1 -- 172.21.15.195:0/1501059861 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f726014ed10 msgr2=0x7f7260152ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:58.764 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.758+0000 7f72671b3700 1 --2- 172.21.15.195:0/1501059861 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f726014ed10 0x7f7260152ec0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7260072ad0 tx=0x7f7254009b20 comp rx=0 tx=0).stop 2023-10-14T11:57:58.764 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.758+0000 7f72671b3700 1 -- 172.21.15.195:0/1501059861 shutdown_connections 2023-10-14T11:57:58.765 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.758+0000 7f72671b3700 1 --2- 172.21.15.195:0/1501059861 >> v2:172.21.15.78:3300/0 conn(0x7f7260154c20 0x7f7260155030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.765 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.758+0000 7f72671b3700 1 --2- 172.21.15.195:0/1501059861 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f726014ed10 0x7f7260152ec0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.765 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.758+0000 7f72671b3700 1 -- 172.21.15.195:0/1501059861 >> 172.21.15.195:0/1501059861 conn(0x7f72600c3a50 msgr2=0x7f72600c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:57:58.765 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.758+0000 7f72671b3700 1 -- 172.21.15.195:0/1501059861 shutdown_connections 2023-10-14T11:57:58.765 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.758+0000 7f72671b3700 1 -- 172.21.15.195:0/1501059861 wait complete. 2023-10-14T11:57:58.768 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.760+0000 7f72671b3700 1 Processor -- start 2023-10-14T11:57:58.768 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.760+0000 7f72671b3700 1 -- start start 2023-10-14T11:57:58.769 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.761+0000 7f72671b3700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f726014ed10 0x7f72601ebfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:58.769 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.761+0000 7f72671b3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7260154c20 0x7f726006ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:58.769 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.762+0000 7f72671b3700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f72601e8b80 0x7f72600ab850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:58.769 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.762+0000 7f72671b3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72601e8760 con 0x7f7260154c20 2023-10-14T11:57:58.769 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.762+0000 7f72671b3700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7260151cb0 con 0x7f726014ed10 2023-10-14T11:57:58.770 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.762+0000 7f72671b3700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f726006b7f0 con 0x7f72601e8b80 2023-10-14T11:57:58.770 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.762+0000 7f7264f4f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f726014ed10 0x7f72601ebfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:58.770 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.762+0000 7f7265750700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f72601e8b80 0x7f72600ab850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:58.771 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.762+0000 7f7264f4f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f726014ed10 0x7f72601ebfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44884/0 (socket says 172.21.15.195:44884) 2023-10-14T11:57:58.771 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.762+0000 7f7264f4f700 1 -- 172.21.15.195:0/749221762 learned_addr learned my addr 172.21.15.195:0/749221762 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-14T11:57:58.771 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.762+0000 7f725ffff700 1 --2- 172.21.15.195:0/749221762 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7260154c20 0x7f726006ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:58.771 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.762+0000 7f7265750700 1 -- 172.21.15.195:0/749221762 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f726014ed10 msgr2=0x7f72601ebfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:58.771 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.762+0000 7f7265750700 1 --2- 172.21.15.195:0/749221762 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f726014ed10 0x7f72601ebfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.772 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.762+0000 7f7265750700 1 -- 172.21.15.195:0/749221762 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7260154c20 msgr2=0x7f726006ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:58.772 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.762+0000 7f7265750700 1 --2- 172.21.15.195:0/749221762 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7260154c20 0x7f726006ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.772 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.762+0000 7f7265750700 1 -- 172.21.15.195:0/749221762 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72540094a0 con 0x7f72601e8b80 2023-10-14T11:57:58.772 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.762+0000 7f7264f4f700 1 --2- 172.21.15.195:0/749221762 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f726014ed10 0x7f72601ebfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T11:57:58.775 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.762+0000 7f7265750700 1 --2- 172.21.15.195:0/749221762 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f72601e8b80 0x7f72600ab850 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f725800eb80 tx=0x7f725800ee90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:57:58.776 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.762+0000 7f725d7fa700 1 -- 172.21.15.195:0/749221762 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725800c940 con 0x7f72601e8b80 2023-10-14T11:57:58.776 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.762+0000 7f725d7fa700 1 -- 172.21.15.195:0/749221762 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f725800f040 con 0x7f72601e8b80 2023-10-14T11:57:58.776 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.762+0000 7f72671b3700 1 -- 172.21.15.195:0/749221762 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72601e80e0 con 0x7f72601e8b80 2023-10-14T11:57:58.777 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.762+0000 7f725d7fa700 1 -- 172.21.15.195:0/749221762 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725800cd00 con 0x7f72601e8b80 2023-10-14T11:57:58.777 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.762+0000 7f72671b3700 1 -- 172.21.15.195:0/749221762 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f726006b510 con 0x7f72601e8b80 2023-10-14T11:57:58.777 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.763+0000 7f725d7fa700 1 -- 172.21.15.195:0/749221762 <== mon.2 v2:172.21.15.195:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7258014780 con 0x7f72601e8b80 2023-10-14T11:57:58.777 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.763+0000 7f725d7fa700 1 -- 172.21.15.195:0/749221762 <== mon.2 v2:172.21.15.195:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f725801dc00 con 0x7f72601e8b80 2023-10-14T11:57:58.777 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.763+0000 7f72671b3700 1 -- 172.21.15.195:0/749221762 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72601e7f40 con 0x7f72601e8b80 2023-10-14T11:57:58.778 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.765+0000 7f725d7fa700 1 -- 172.21.15.195:0/749221762 <== mon.2 v2:172.21.15.195:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f72601e7f40 con 0x7f72601e8b80 2023-10-14T11:57:58.850 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.848+0000 7f72671b3700 1 -- 172.21.15.195:0/749221762 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_command({"prefix": "osd new", "uuid": "2f75ded1-e9e2-40e7-99ae-45e1e425e1f0", "id": 3} v 0) v1 -- 0x7f72601e88a0 con 0x7f72601e8b80 2023-10-14T11:57:58.853 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.851+0000 7f725d7fa700 1 -- 172.21.15.195:0/749221762 <== mon.2 v2:172.21.15.195:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2f75ded1-e9e2-40e7-99ae-45e1e425e1f0", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7258018020 con 0x7f72601e8b80 2023-10-14T11:57:58.853 INFO:teuthology.orchestra.run.smithi195.stdout:3 2023-10-14T11:57:58.854 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.852+0000 7f72671b3700 1 -- 172.21.15.195:0/749221762 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f72601e8b80 msgr2=0x7f72600ab850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:58.855 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.852+0000 7f72671b3700 1 --2- 172.21.15.195:0/749221762 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f72601e8b80 0x7f72600ab850 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f725800eb80 tx=0x7f725800ee90 comp rx=0 tx=0).stop 2023-10-14T11:57:58.855 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.852+0000 7f72671b3700 1 -- 172.21.15.195:0/749221762 shutdown_connections 2023-10-14T11:57:58.855 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.852+0000 7f72671b3700 1 --2- 172.21.15.195:0/749221762 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f72601e8b80 0x7f72600ab850 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.855 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.852+0000 7f72671b3700 1 --2- 172.21.15.195:0/749221762 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f726014ed10 0x7f72601ebfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.855 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.852+0000 7f72671b3700 1 --2- 172.21.15.195:0/749221762 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7260154c20 0x7f726006ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.856 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.852+0000 7f72671b3700 1 -- 172.21.15.195:0/749221762 >> 172.21.15.195:0/749221762 conn(0x7f72600c3a50 msgr2=0x7f72600c6710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:57:58.856 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.852+0000 7f72671b3700 1 -- 172.21.15.195:0/749221762 shutdown_connections 2023-10-14T11:57:58.856 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.852+0000 7f72671b3700 1 -- 172.21.15.195:0/749221762 wait complete. 2023-10-14T11:57:58.866 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 5130b432-8358-46cb-8a31-3064c976f6d8 4 2023-10-14T11:57:58.987 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.985+0000 7f6f36a15700 1 Processor -- start 2023-10-14T11:57:58.988 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.985+0000 7f6f36a15700 1 -- start start 2023-10-14T11:57:58.993 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.986+0000 7f6f36a15700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6f300c2a50 0x7f6f30158b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:58.993 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.986+0000 7f6f36a15700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6f300c11a0 0x7f6f301552f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:58.994 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.986+0000 7f6f36a15700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6f30155830 0x7f6f300c6770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:58.994 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.986+0000 7f6f36a15700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f30159050 con 0x7f6f300c2a50 2023-10-14T11:57:58.994 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.986+0000 7f6f36a15700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f30154b10 con 0x7f6f300c11a0 2023-10-14T11:57:58.995 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.986+0000 7f6f36a15700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f30072220 con 0x7f6f30155830 2023-10-14T11:57:58.995 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.986+0000 7f6f2f7fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6f300c11a0 0x7f6f301552f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:58.995 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.986+0000 7f6f34fb2700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6f30155830 0x7f6f300c6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:58.995 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.986+0000 7f6f2f7fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6f300c11a0 0x7f6f301552f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44892/0 (socket says 172.21.15.195:44892) 2023-10-14T11:57:58.996 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.986+0000 7f6f34fb2700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6f30155830 0x7f6f300c6770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.195:39478/0 (socket says 172.21.15.195:39478) 2023-10-14T11:57:58.996 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.986+0000 7f6f2f7fe700 1 -- 172.21.15.195:0/474552536 learned_addr learned my addr 172.21.15.195:0/474552536 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-14T11:57:58.996 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.986+0000 7f6f2ffff700 1 --2- 172.21.15.195:0/474552536 >> v2:172.21.15.78:3300/0 conn(0x7f6f300c2a50 0x7f6f30158b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:58.996 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.986+0000 7f6f34fb2700 1 -- 172.21.15.195:0/474552536 >> v2:172.21.15.195:3300/0 conn(0x7f6f300c11a0 msgr2=0x7f6f301552f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:57:58.997 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.986+0000 7f6f34fb2700 1 --2- 172.21.15.195:0/474552536 >> v2:172.21.15.195:3300/0 conn(0x7f6f300c11a0 0x7f6f301552f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.997 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.986+0000 7f6f34fb2700 1 -- 172.21.15.195:0/474552536 >> v2:172.21.15.78:3300/0 conn(0x7f6f300c2a50 msgr2=0x7f6f30158b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:57:58.997 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.986+0000 7f6f34fb2700 1 --2- 172.21.15.195:0/474552536 >> v2:172.21.15.78:3300/0 conn(0x7f6f300c2a50 0x7f6f30158b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:58.998 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.986+0000 7f6f34fb2700 1 -- 172.21.15.195:0/474552536 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f30072500 con 0x7f6f30155830 2023-10-14T11:57:58.998 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.986+0000 7f6f2ffff700 1 --2- 172.21.15.195:0/474552536 >> v2:172.21.15.78:3300/0 conn(0x7f6f300c2a50 0x7f6f30158b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:57:58.998 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.986+0000 7f6f2f7fe700 1 --2- 172.21.15.195:0/474552536 >> v2:172.21.15.195:3300/0 conn(0x7f6f300c11a0 0x7f6f301552f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T11:57:58.998 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.986+0000 7f6f34fb2700 1 --2- 172.21.15.195:0/474552536 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6f30155830 0x7f6f300c6770 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6f300c4340 tx=0x7f6f2800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d599b00d638737fe server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:57:58.999 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.987+0000 7f6f2effd700 1 -- 172.21.15.195:0/474552536 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f2800e070 con 0x7f6f30155830 2023-10-14T11:57:58.999 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.987+0000 7f6f2effd700 1 -- 172.21.15.195:0/474552536 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f2800bb20 con 0x7f6f30155830 2023-10-14T11:57:58.999 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.987+0000 7f6f2effd700 1 -- 172.21.15.195:0/474552536 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f28012430 con 0x7f6f30155830 2023-10-14T11:57:59.000 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.987+0000 7f6f36a15700 1 -- 172.21.15.195:0/474552536 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6f30155830 msgr2=0x7f6f300c6770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:59.000 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.987+0000 7f6f36a15700 1 --2- 172.21.15.195:0/474552536 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6f30155830 0x7f6f300c6770 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6f300c4340 tx=0x7f6f2800b330 comp rx=0 tx=0).stop 2023-10-14T11:57:59.000 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.987+0000 7f6f36a15700 1 -- 172.21.15.195:0/474552536 shutdown_connections 2023-10-14T11:57:59.000 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.987+0000 7f6f36a15700 1 --2- 172.21.15.195:0/474552536 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6f30155830 0x7f6f300c6770 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:59.001 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.987+0000 7f6f36a15700 1 --2- 172.21.15.195:0/474552536 >> v2:172.21.15.78:3300/0 conn(0x7f6f300c2a50 0x7f6f30158b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:59.001 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.987+0000 7f6f36a15700 1 --2- 172.21.15.195:0/474552536 >> v2:172.21.15.195:3300/0 conn(0x7f6f300c11a0 0x7f6f301552f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:59.001 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.987+0000 7f6f36a15700 1 -- 172.21.15.195:0/474552536 >> 172.21.15.195:0/474552536 conn(0x7f6f300a8410 msgr2=0x7f6f300a8820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:57:59.001 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.987+0000 7f6f36a15700 1 -- 172.21.15.195:0/474552536 shutdown_connections 2023-10-14T11:57:59.001 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.987+0000 7f6f36a15700 1 -- 172.21.15.195:0/474552536 wait complete. 2023-10-14T11:57:59.002 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.988+0000 7f6f36a15700 1 Processor -- start 2023-10-14T11:57:59.002 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.989+0000 7f6f36a15700 1 -- start start 2023-10-14T11:57:59.002 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.989+0000 7f6f36a15700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6f300c11a0 0x7f6f300aba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:59.002 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.989+0000 7f6f36a15700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f300c2a50 0x7f6f30157370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:59.002 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.989+0000 7f6f36a15700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6f301ee590 0x7f6f300a7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:59.003 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.989+0000 7f6f36a15700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f301ee170 con 0x7f6f300c2a50 2023-10-14T11:57:59.003 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.989+0000 7f6f36a15700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f300ac4e0 con 0x7f6f301ee590 2023-10-14T11:57:59.003 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.989+0000 7f6f36a15700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f300abf60 con 0x7f6f300c11a0 2023-10-14T11:57:59.003 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.989+0000 7f6f2ffff700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6f300c11a0 0x7f6f300aba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:59.003 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.989+0000 7f6f2ffff700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6f300c11a0 0x7f6f300aba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.195:39480/0 (socket says 172.21.15.195:39480) 2023-10-14T11:57:59.004 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.989+0000 7f6f2ffff700 1 -- 172.21.15.195:0/3038808678 learned_addr learned my addr 172.21.15.195:0/3038808678 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-14T11:57:59.004 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.989+0000 7f6f34fb2700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6f301ee590 0x7f6f300a7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:59.004 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.990+0000 7f6f2f7fe700 1 --2- 172.21.15.195:0/3038808678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f300c2a50 0x7f6f30157370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:59.004 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.990+0000 7f6f2ffff700 1 -- 172.21.15.195:0/3038808678 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6f301ee590 msgr2=0x7f6f300a7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:59.004 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.990+0000 7f6f2ffff700 1 --2- 172.21.15.195:0/3038808678 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6f301ee590 0x7f6f300a7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:59.005 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.990+0000 7f6f2ffff700 1 -- 172.21.15.195:0/3038808678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f300c2a50 msgr2=0x7f6f30157370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:59.005 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.990+0000 7f6f2ffff700 1 --2- 172.21.15.195:0/3038808678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f300c2a50 0x7f6f30157370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:59.005 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.990+0000 7f6f2ffff700 1 -- 172.21.15.195:0/3038808678 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f24009550 con 0x7f6f300c11a0 2023-10-14T11:57:59.005 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.990+0000 7f6f34fb2700 1 --2- 172.21.15.195:0/3038808678 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6f301ee590 0x7f6f300a7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T11:57:59.005 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.990+0000 7f6f2ffff700 1 --2- 172.21.15.195:0/3038808678 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6f300c11a0 0x7f6f300aba20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6f2000e810 tx=0x7f6f2000eb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:57:59.005 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.990+0000 7f6f2cff9700 1 -- 172.21.15.195:0/3038808678 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f20004230 con 0x7f6f300c11a0 2023-10-14T11:57:59.006 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.990+0000 7f6f2cff9700 1 -- 172.21.15.195:0/3038808678 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f200043b0 con 0x7f6f300c11a0 2023-10-14T11:57:59.006 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.990+0000 7f6f36a15700 1 -- 172.21.15.195:0/3038808678 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f28009a70 con 0x7f6f300c11a0 2023-10-14T11:57:59.006 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.990+0000 7f6f2cff9700 1 -- 172.21.15.195:0/3038808678 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f200038e0 con 0x7f6f300c11a0 2023-10-14T11:57:59.006 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.990+0000 7f6f36a15700 1 -- 172.21.15.195:0/3038808678 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f301edab0 con 0x7f6f300c11a0 2023-10-14T11:57:59.006 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.990+0000 7f6f2cff9700 1 -- 172.21.15.195:0/3038808678 <== mon.2 v2:172.21.15.195:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f6f20003d40 con 0x7f6f300c11a0 2023-10-14T11:57:59.007 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.990+0000 7f6f2cff9700 1 -- 172.21.15.195:0/3038808678 <== mon.2 v2:172.21.15.195:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f6f20013c30 con 0x7f6f300c11a0 2023-10-14T11:57:59.007 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.991+0000 7f6f36a15700 1 -- 172.21.15.195:0/3038808678 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f301ee3e0 con 0x7f6f300c11a0 2023-10-14T11:57:59.007 INFO:tasks.ceph.mgr.y.smithi078.stderr:2023-10-14T11:57:58.994+0000 7ff4d96d6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T11:57:59.008 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:58.993+0000 7f6f2cff9700 1 -- 172.21.15.195:0/3038808678 <== mon.2 v2:172.21.15.195:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f6f20020060 con 0x7f6f300c11a0 2023-10-14T11:57:59.054 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-10-14T11:57:59.052+0000 7f94eb2a9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T11:57:59.078 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.076+0000 7f6f36a15700 1 -- 172.21.15.195:0/3038808678 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_command({"prefix": "osd new", "uuid": "5130b432-8358-46cb-8a31-3064c976f6d8", "id": 4} v 0) v1 -- 0x7f6f301ed910 con 0x7f6f300c11a0 2023-10-14T11:57:59.083 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.082+0000 7f6f2cff9700 1 -- 172.21.15.195:0/3038808678 <== mon.2 v2:172.21.15.195:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5130b432-8358-46cb-8a31-3064c976f6d8", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6f301ed910 con 0x7f6f300c11a0 2023-10-14T11:57:59.084 INFO:teuthology.orchestra.run.smithi195.stdout:4 2023-10-14T11:57:59.085 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.083+0000 7f6f36a15700 1 -- 172.21.15.195:0/3038808678 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6f300c11a0 msgr2=0x7f6f300aba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:59.085 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.083+0000 7f6f36a15700 1 --2- 172.21.15.195:0/3038808678 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6f300c11a0 0x7f6f300aba20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6f2000e810 tx=0x7f6f2000eb20 comp rx=0 tx=0).stop 2023-10-14T11:57:59.085 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.083+0000 7f6f36a15700 1 -- 172.21.15.195:0/3038808678 shutdown_connections 2023-10-14T11:57:59.086 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.083+0000 7f6f36a15700 1 --2- 172.21.15.195:0/3038808678 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6f301ee590 0x7f6f300a7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:59.086 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.083+0000 7f6f36a15700 1 --2- 172.21.15.195:0/3038808678 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6f300c11a0 0x7f6f300aba20 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:59.086 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.083+0000 7f6f36a15700 1 --2- 172.21.15.195:0/3038808678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f300c2a50 0x7f6f30157370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:59.087 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.083+0000 7f6f36a15700 1 -- 172.21.15.195:0/3038808678 >> 172.21.15.195:0/3038808678 conn(0x7f6f300a8410 msgr2=0x7f6f30153f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:57:59.087 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.083+0000 7f6f36a15700 1 -- 172.21.15.195:0/3038808678 shutdown_connections 2023-10-14T11:57:59.087 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.083+0000 7f6f36a15700 1 -- 172.21.15.195:0/3038808678 wait complete. 2023-10-14T11:57:59.098 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 60e234a9-a506-4e40-9c19-49a58f3cf28c 5 2023-10-14T11:57:59.221 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.219+0000 7f8694b9f700 1 Processor -- start 2023-10-14T11:57:59.222 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.220+0000 7f8694b9f700 1 -- start start 2023-10-14T11:57:59.225 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.220+0000 7f8694b9f700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f86900c6200 0x7f8690151ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:59.225 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.220+0000 7f8694b9f700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8690071180 con 0x7f86900c2370 2023-10-14T11:57:59.225 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.220+0000 7f8694b9f700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86900c6610 con 0x7f86900a5eb0 2023-10-14T11:57:59.226 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.220+0000 7f8694b9f700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86900ba690 con 0x7f86900c6200 2023-10-14T11:57:59.226 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.220+0000 7f868effd700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f86900a5eb0 0x7f86901515d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:39634/0 (socket says 172.21.15.195:39634) 2023-10-14T11:57:59.226 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.220+0000 7f868effd700 1 -- 172.21.15.195:0/2172930384 learned_addr learned my addr 172.21.15.195:0/2172930384 (peer_addr_for_me v1:172.21.15.195:0/0) 2023-10-14T11:57:59.226 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.221+0000 7f868ffff700 1 --2- 172.21.15.195:0/2172930384 >> v2:172.21.15.78:3300/0 conn(0x7f86900c6200 0x7f8690151ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:59.227 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.221+0000 7f868e7fc700 1 -- 172.21.15.195:0/2172930384 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4011179779 0 0) 0x7f86900c6610 con 0x7f86900a5eb0 2023-10-14T11:57:59.227 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.221+0000 7f868e7fc700 1 -- 172.21.15.195:0/2172930384 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86780035b0 con 0x7f86900a5eb0 2023-10-14T11:57:59.227 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.221+0000 7f868ffff700 1 -- 172.21.15.195:0/2172930384 >> v1:172.21.15.195:6789/0 conn(0x7f86900a5eb0 legacy=0x7f86901515d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:59.227 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.221+0000 7f868ffff700 1 -- 172.21.15.195:0/2172930384 >> v1:172.21.15.78:6789/0 conn(0x7f86900c2370 legacy=0x7f86900c5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:59.227 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.221+0000 7f868e7fc700 1 -- 172.21.15.195:0/2172930384 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1621871588 0 0) 0x7f8690071180 con 0x7f86900c2370 2023-10-14T11:57:59.228 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.221+0000 7f868ffff700 1 -- 172.21.15.195:0/2172930384 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86900c18e0 con 0x7f86900c6200 2023-10-14T11:57:59.228 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.221+0000 7f868ffff700 1 --2- 172.21.15.195:0/2172930384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86900c6200 0x7f8690151ce0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8688009d90 tx=0x7f868800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d61a20b74262035 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:57:59.228 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.221+0000 7f868e7fc700 1 -- 172.21.15.195:0/2172930384 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868800e070 con 0x7f86900c6200 2023-10-14T11:57:59.228 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.221+0000 7f868e7fc700 1 -- 172.21.15.195:0/2172930384 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f868800bb20 con 0x7f86900c6200 2023-10-14T11:57:59.228 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.221+0000 7f868e7fc700 1 -- 172.21.15.195:0/2172930384 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868800be90 con 0x7f86900c6200 2023-10-14T11:57:59.229 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.222+0000 7f8694b9f700 1 -- 172.21.15.195:0/2172930384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86900c6200 msgr2=0x7f8690151ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:59.229 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.222+0000 7f8694b9f700 1 --2- 172.21.15.195:0/2172930384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86900c6200 0x7f8690151ce0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8688009d90 tx=0x7f868800b330 comp rx=0 tx=0).stop 2023-10-14T11:57:59.229 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.222+0000 7f8694b9f700 1 -- 172.21.15.195:0/2172930384 shutdown_connections 2023-10-14T11:57:59.229 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.222+0000 7f8694b9f700 1 --2- 172.21.15.195:0/2172930384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86900c6200 0x7f8690151ce0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:59.230 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.222+0000 7f8694b9f700 1 -- 172.21.15.195:0/2172930384 >> 172.21.15.195:0/2172930384 conn(0x7f86900c6de0 msgr2=0x7f86900c71c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:57:59.230 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.222+0000 7f8694b9f700 1 -- 172.21.15.195:0/2172930384 shutdown_connections 2023-10-14T11:57:59.230 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.222+0000 7f8694b9f700 1 -- 172.21.15.195:0/2172930384 wait complete. 2023-10-14T11:57:59.230 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.223+0000 7f8694b9f700 1 Processor -- start 2023-10-14T11:57:59.231 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.223+0000 7f8694b9f700 1 -- start start 2023-10-14T11:57:59.233 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.224+0000 7f8694b9f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86900a5eb0 0x7f86901eaa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:59.233 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.224+0000 7f8694b9f700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f86900c2370 0x7f86900abda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:59.234 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.224+0000 7f8694b9f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86900c6200 0x7f86900bd360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:59.234 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.224+0000 7f8694b9f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86900c0650 con 0x7f86900a5eb0 2023-10-14T11:57:59.234 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.224+0000 7f8694b9f700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86900be8f0 con 0x7f86900c6200 2023-10-14T11:57:59.234 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.224+0000 7f8694b9f700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86900a6730 con 0x7f86900c2370 2023-10-14T11:57:59.234 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.224+0000 7f868ffff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86900c6200 0x7f86900bd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:59.235 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.224+0000 7f868effd700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f86900c2370 0x7f86900abda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:59.235 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.224+0000 7f868ffff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86900c6200 0x7f86900bd360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44912/0 (socket says 172.21.15.195:44912) 2023-10-14T11:57:59.235 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.224+0000 7f868ffff700 1 -- 172.21.15.195:0/855743521 learned_addr learned my addr 172.21.15.195:0/855743521 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-14T11:57:59.235 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.224+0000 7f868f7fe700 1 --2- 172.21.15.195:0/855743521 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86900a5eb0 0x7f86901eaa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:59.236 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.224+0000 7f868ffff700 1 -- 172.21.15.195:0/855743521 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f86900c2370 msgr2=0x7f86900abda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:59.236 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.224+0000 7f868ffff700 1 --2- 172.21.15.195:0/855743521 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f86900c2370 0x7f86900abda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:59.236 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.224+0000 7f868ffff700 1 -- 172.21.15.195:0/855743521 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86900a5eb0 msgr2=0x7f86901eaa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:59.236 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.225+0000 7f868ffff700 1 --2- 172.21.15.195:0/855743521 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86900a5eb0 0x7f86901eaa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:59.236 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.225+0000 7f868ffff700 1 -- 172.21.15.195:0/855743521 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8688009a70 con 0x7f86900c6200 2023-10-14T11:57:59.237 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.225+0000 7f868effd700 1 --2- 172.21.15.195:0/855743521 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f86900c2370 0x7f86900abda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T11:57:59.237 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.225+0000 7f868ffff700 1 --2- 172.21.15.195:0/855743521 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86900c6200 0x7f86900bd360 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f86900c52e0 tx=0x7f8688004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:57:59.237 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.225+0000 7f8677fff700 1 -- 172.21.15.195:0/855743521 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8688004670 con 0x7f86900c6200 2023-10-14T11:57:59.237 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.225+0000 7f8677fff700 1 -- 172.21.15.195:0/855743521 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8688021070 con 0x7f86900c6200 2023-10-14T11:57:59.237 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.225+0000 7f8694b9f700 1 -- 172.21.15.195:0/855743521 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86900a6a10 con 0x7f86900c6200 2023-10-14T11:57:59.238 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.225+0000 7f8677fff700 1 -- 172.21.15.195:0/855743521 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86880098f0 con 0x7f86900c6200 2023-10-14T11:57:59.238 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.225+0000 7f8694b9f700 1 -- 172.21.15.195:0/855743521 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86900a84a0 con 0x7f86900c6200 2023-10-14T11:57:59.238 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.225+0000 7f8677fff700 1 -- 172.21.15.195:0/855743521 <== mon.1 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f8688023030 con 0x7f86900c6200 2023-10-14T11:57:59.238 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.225+0000 7f8677fff700 1 -- 172.21.15.195:0/855743521 <== mon.1 v2:172.21.15.195:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f868802f9b0 con 0x7f86900c6200 2023-10-14T11:57:59.239 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.226+0000 7f8694b9f700 1 -- 172.21.15.195:0/855743521 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86900c55b0 con 0x7f86900c6200 2023-10-14T11:57:59.239 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.228+0000 7f8677fff700 1 -- 172.21.15.195:0/855743521 <== mon.1 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f8688045b40 con 0x7f86900c6200 2023-10-14T11:57:59.312 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.310+0000 7f8694b9f700 1 -- 172.21.15.195:0/855743521 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd new", "uuid": "60e234a9-a506-4e40-9c19-49a58f3cf28c", "id": 5} v 0) v1 -- 0x7f86900a6ba0 con 0x7f86900c6200 2023-10-14T11:57:59.317 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.315+0000 7f8677fff700 1 -- 172.21.15.195:0/855743521 <== mon.1 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "60e234a9-a506-4e40-9c19-49a58f3cf28c", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8688034030 con 0x7f86900c6200 2023-10-14T11:57:59.317 INFO:teuthology.orchestra.run.smithi195.stdout:5 2023-10-14T11:57:59.319 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.316+0000 7f8694b9f700 1 -- 172.21.15.195:0/855743521 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86900c6200 msgr2=0x7f86900bd360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:59.319 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.316+0000 7f8694b9f700 1 --2- 172.21.15.195:0/855743521 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86900c6200 0x7f86900bd360 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f86900c52e0 tx=0x7f8688004ad0 comp rx=0 tx=0).stop 2023-10-14T11:57:59.319 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.316+0000 7f8694b9f700 1 -- 172.21.15.195:0/855743521 shutdown_connections 2023-10-14T11:57:59.319 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.316+0000 7f8694b9f700 1 --2- 172.21.15.195:0/855743521 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86900c6200 0x7f86900bd360 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:59.320 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.316+0000 7f8694b9f700 1 --2- 172.21.15.195:0/855743521 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86900a5eb0 0x7f86901eaa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:59.320 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.316+0000 7f8694b9f700 1 --2- 172.21.15.195:0/855743521 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f86900c2370 0x7f86900abda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:59.320 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.316+0000 7f8694b9f700 1 -- 172.21.15.195:0/855743521 >> 172.21.15.195:0/855743521 conn(0x7f86900c6de0 msgr2=0x7f8690151a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:57:59.320 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.317+0000 7f8694b9f700 1 -- 172.21.15.195:0/855743521 shutdown_connections 2023-10-14T11:57:59.321 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.317+0000 7f8694b9f700 1 -- 172.21.15.195:0/855743521 wait complete. 2023-10-14T11:57:59.330 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new df7d2dd6-26b6-4eb1-96e5-94d1f3955b33 6 2023-10-14T11:57:59.474 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.472+0000 7fb657814700 1 Processor -- start 2023-10-14T11:57:59.476 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.473+0000 7fb657814700 1 -- start start 2023-10-14T11:57:59.477 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.474+0000 7fb657814700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb6500c0830 0x7fb650072a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:59.477 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.474+0000 7fb657814700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb6500706b0 0x7fb6501548c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:59.477 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.474+0000 7fb657814700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6500c18b0 con 0x7fb6500a7ba0 2023-10-14T11:57:59.477 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.474+0000 7fb657814700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6500c15d0 con 0x7fb6500c0830 2023-10-14T11:57:59.478 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.474+0000 7fb657814700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6500c3f30 con 0x7fb6500706b0 2023-10-14T11:57:59.478 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.474+0000 7fb654daf700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb6500c0830 0x7fb650072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:59.478 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.474+0000 7fb6555b0700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb6500a7ba0 0x7fb6500c54a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.195:35694/0 (socket says 172.21.15.195:35694) 2023-10-14T11:57:59.478 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.474+0000 7fb6555b0700 1 -- 172.21.15.195:0/772538361 learned_addr learned my addr 172.21.15.195:0/772538361 (peer_addr_for_me v1:172.21.15.195:0/0) 2023-10-14T11:57:59.478 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.474+0000 7fb654daf700 1 -- 172.21.15.195:0/772538361 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb6500706b0 msgr2=0x7fb6501548c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-14T11:57:59.479 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.474+0000 7fb654daf700 1 --2- 172.21.15.195:0/772538361 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb6500706b0 0x7fb6501548c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:59.479 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.474+0000 7fb654daf700 1 -- 172.21.15.195:0/772538361 >> v1:172.21.15.78:6789/0 conn(0x7fb6500a7ba0 legacy=0x7fb6500c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:59.479 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.474+0000 7fb654daf700 1 -- 172.21.15.195:0/772538361 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6500c4210 con 0x7fb6500c0830 2023-10-14T11:57:59.479 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.474+0000 7fb647fff700 1 -- 172.21.15.195:0/772538361 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 995988230 0 0) 0x7fb6500c18b0 con 0x7fb6500a7ba0 2023-10-14T11:57:59.479 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.475+0000 7fb654daf700 1 --2- 172.21.15.195:0/772538361 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6500c0830 0x7fb650072a70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb640009840 tx=0x7fb640009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6c5689ab99b0b02 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:57:59.481 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.475+0000 7fb647fff700 1 -- 172.21.15.195:0/772538361 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb640011830 con 0x7fb6500c0830 2023-10-14T11:57:59.481 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.475+0000 7fb647fff700 1 -- 172.21.15.195:0/772538361 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb640011990 con 0x7fb6500c0830 2023-10-14T11:57:59.481 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.475+0000 7fb647fff700 1 -- 172.21.15.195:0/772538361 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb640011d20 con 0x7fb6500c0830 2023-10-14T11:57:59.481 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.476+0000 7fb657814700 1 -- 172.21.15.195:0/772538361 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6500c0830 msgr2=0x7fb650072a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:59.481 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.476+0000 7fb657814700 1 --2- 172.21.15.195:0/772538361 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6500c0830 0x7fb650072a70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb640009840 tx=0x7fb640009b50 comp rx=0 tx=0).stop 2023-10-14T11:57:59.482 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.476+0000 7fb657814700 1 -- 172.21.15.195:0/772538361 shutdown_connections 2023-10-14T11:57:59.482 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.476+0000 7fb657814700 1 --2- 172.21.15.195:0/772538361 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb6500706b0 0x7fb6501548c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:59.482 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.476+0000 7fb657814700 1 --2- 172.21.15.195:0/772538361 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6500c0830 0x7fb650072a70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:59.482 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.476+0000 7fb657814700 1 -- 172.21.15.195:0/772538361 >> 172.21.15.195:0/772538361 conn(0x7fb65006aab0 msgr2=0x7fb65006aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:57:59.482 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.476+0000 7fb657814700 1 -- 172.21.15.195:0/772538361 shutdown_connections 2023-10-14T11:57:59.483 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.476+0000 7fb657814700 1 -- 172.21.15.195:0/772538361 wait complete. 2023-10-14T11:57:59.483 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.477+0000 7fb657814700 1 Processor -- start 2023-10-14T11:57:59.484 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.478+0000 7fb657814700 1 -- start start 2023-10-14T11:57:59.484 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.478+0000 7fb657814700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb6500706b0 0x7fb6501f15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:59.485 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.478+0000 7fb657814700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6500a7ba0 0x7fb6501f1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:59.485 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.478+0000 7fb657814700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6500c0830 0x7fb65014ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:59.485 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.479+0000 7fb657814700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6501f21e0 con 0x7fb6500c0830 2023-10-14T11:57:59.485 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.479+0000 7fb657814700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6501efe00 con 0x7fb6500a7ba0 2023-10-14T11:57:59.486 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.479+0000 7fb657814700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6501ef4d0 con 0x7fb6500706b0 2023-10-14T11:57:59.486 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.479+0000 7fb655db1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6500c0830 0x7fb65014ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:59.486 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.479+0000 7fb6555b0700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb6500706b0 0x7fb6501f15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:59.486 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.479+0000 7fb654daf700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6500a7ba0 0x7fb6501f1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:59.486 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.479+0000 7fb655db1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6500c0830 0x7fb65014ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.195:59998/0 (socket says 172.21.15.195:59998) 2023-10-14T11:57:59.487 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.479+0000 7fb655db1700 1 -- 172.21.15.195:0/796673475 learned_addr learned my addr 172.21.15.195:0/796673475 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-14T11:57:59.487 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.479+0000 7fb655db1700 1 -- 172.21.15.195:0/796673475 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb6500706b0 msgr2=0x7fb6501f15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:59.487 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.479+0000 7fb655db1700 1 --2- 172.21.15.195:0/796673475 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb6500706b0 0x7fb6501f15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:59.487 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.479+0000 7fb655db1700 1 -- 172.21.15.195:0/796673475 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6500a7ba0 msgr2=0x7fb6501f1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:59.487 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.479+0000 7fb655db1700 1 --2- 172.21.15.195:0/796673475 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6500a7ba0 0x7fb6501f1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:59.488 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.479+0000 7fb655db1700 1 -- 172.21.15.195:0/796673475 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6400094a0 con 0x7fb6500c0830 2023-10-14T11:57:59.488 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.479+0000 7fb6555b0700 1 --2- 172.21.15.195:0/796673475 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb6500706b0 0x7fb6501f15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T11:57:59.488 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.479+0000 7fb654daf700 1 --2- 172.21.15.195:0/796673475 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6500a7ba0 0x7fb6501f1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T11:57:59.488 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.479+0000 7fb655db1700 1 --2- 172.21.15.195:0/796673475 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6500c0830 0x7fb65014ea40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb64c00b550 tx=0x7fb64c00b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:57:59.488 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.480+0000 7fb645ffb700 1 -- 172.21.15.195:0/796673475 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64c00bee0 con 0x7fb6500c0830 2023-10-14T11:57:59.488 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.480+0000 7fb657814700 1 -- 172.21.15.195:0/796673475 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6501ef7b0 con 0x7fb6500c0830 2023-10-14T11:57:59.489 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.480+0000 7fb657814700 1 -- 172.21.15.195:0/796673475 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb650153000 con 0x7fb6500c0830 2023-10-14T11:57:59.489 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.480+0000 7fb645ffb700 1 -- 172.21.15.195:0/796673475 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb64c033640 con 0x7fb6500c0830 2023-10-14T11:57:59.489 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.480+0000 7fb645ffb700 1 -- 172.21.15.195:0/796673475 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64c0339f0 con 0x7fb6500c0830 2023-10-14T11:57:59.489 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.480+0000 7fb645ffb700 1 -- 172.21.15.195:0/796673475 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb64c03c400 con 0x7fb6500c0830 2023-10-14T11:57:59.489 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.481+0000 7fb645ffb700 1 -- 172.21.15.195:0/796673475 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7fb64c03cca0 con 0x7fb6500c0830 2023-10-14T11:57:59.490 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.481+0000 7fb657814700 1 -- 172.21.15.195:0/796673475 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6300052f0 con 0x7fb6500c0830 2023-10-14T11:57:59.490 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.485+0000 7fb645ffb700 1 -- 172.21.15.195:0/796673475 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fb64c05bb40 con 0x7fb6500c0830 2023-10-14T11:57:59.589 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.587+0000 7fb657814700 1 -- 172.21.15.195:0/796673475 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd new", "uuid": "df7d2dd6-26b6-4eb1-96e5-94d1f3955b33", "id": 6} v 0) v1 -- 0x7fb630005640 con 0x7fb6500c0830 2023-10-14T11:57:59.593 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.591+0000 7fb645ffb700 1 -- 172.21.15.195:0/796673475 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "df7d2dd6-26b6-4eb1-96e5-94d1f3955b33", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb64c04a070 con 0x7fb6500c0830 2023-10-14T11:57:59.593 INFO:teuthology.orchestra.run.smithi195.stdout:6 2023-10-14T11:57:59.594 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.592+0000 7fb657814700 1 -- 172.21.15.195:0/796673475 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6500c0830 msgr2=0x7fb65014ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:59.595 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.592+0000 7fb657814700 1 --2- 172.21.15.195:0/796673475 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6500c0830 0x7fb65014ea40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb64c00b550 tx=0x7fb64c00b860 comp rx=0 tx=0).stop 2023-10-14T11:57:59.595 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.592+0000 7fb657814700 1 -- 172.21.15.195:0/796673475 shutdown_connections 2023-10-14T11:57:59.595 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.592+0000 7fb657814700 1 --2- 172.21.15.195:0/796673475 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6500c0830 0x7fb65014ea40 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:59.595 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.592+0000 7fb657814700 1 --2- 172.21.15.195:0/796673475 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb6500706b0 0x7fb6501f15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:59.595 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.592+0000 7fb657814700 1 --2- 172.21.15.195:0/796673475 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6500a7ba0 0x7fb6501f1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:59.596 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.592+0000 7fb657814700 1 -- 172.21.15.195:0/796673475 >> 172.21.15.195:0/796673475 conn(0x7fb65006aab0 msgr2=0x7fb65006ae90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:57:59.596 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.592+0000 7fb657814700 1 -- 172.21.15.195:0/796673475 shutdown_connections 2023-10-14T11:57:59.596 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.592+0000 7fb657814700 1 -- 172.21.15.195:0/796673475 wait complete. 2023-10-14T11:57:59.606 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 47a9d4de-f56d-4aa4-9900-c4a9cf9b2f97 7 2023-10-14T11:57:59.635 INFO:tasks.ceph.mgr.y.smithi078.stderr:2023-10-14T11:57:59.633+0000 7ff4d96d6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T11:57:59.703 INFO:tasks.ceph.mgr.y.smithi078.stderr:2023-10-14T11:57:59.701+0000 7ff4d96d6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T11:57:59.729 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.727+0000 7f944e1f0700 1 Processor -- start 2023-10-14T11:57:59.729 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.728+0000 7f944e1f0700 1 -- start start 2023-10-14T11:57:59.731 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.728+0000 7f944e1f0700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f94480c4540 0x7f94480c02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:59.732 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.728+0000 7f944e1f0700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f944806b500 0x7f94480c0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:59.732 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.728+0000 7f944e1f0700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f944806ebe0 con 0x7f944813dbe0 2023-10-14T11:57:59.732 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.728+0000 7f944e1f0700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f944806d330 con 0x7f94480c4540 2023-10-14T11:57:59.732 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.728+0000 7f944e1f0700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94480c0d60 con 0x7f944806b500 2023-10-14T11:57:59.733 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.728+0000 7f9446ffd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f94480c4540 0x7f94480c02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:59.733 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.728+0000 7f9447fff700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f944806b500 0x7f94480c0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:59.733 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.728+0000 7f9446ffd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f94480c4540 0x7f94480c02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44926/0 (socket says 172.21.15.195:44926) 2023-10-14T11:57:59.733 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.728+0000 7f9446ffd700 1 -- 172.21.15.195:0/4001689632 learned_addr learned my addr 172.21.15.195:0/4001689632 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-14T11:57:59.733 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.729+0000 7f9446ffd700 1 -- 172.21.15.195:0/4001689632 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f944806b500 msgr2=0x7f94480c0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:57:59.734 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.729+0000 7f9446ffd700 1 --2- 172.21.15.195:0/4001689632 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f944806b500 0x7f94480c0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:59.734 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.729+0000 7f9446ffd700 1 -- 172.21.15.195:0/4001689632 >> v1:172.21.15.78:6789/0 conn(0x7f944813dbe0 legacy=0x7f944806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:59.734 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.729+0000 7f9446ffd700 1 -- 172.21.15.195:0/4001689632 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94481558c0 con 0x7f94480c4540 2023-10-14T11:57:59.734 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.729+0000 7f9446ffd700 1 --2- 172.21.15.195:0/4001689632 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f94480c4540 0x7f94480c02e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f943c0098f0 tx=0x7f943c009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2853eff11c0c5b54 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:57:59.734 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.729+0000 7f94467fc700 1 -- 172.21.15.195:0/4001689632 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f943c011830 con 0x7f94480c4540 2023-10-14T11:57:59.735 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.729+0000 7f94467fc700 1 -- 172.21.15.195:0/4001689632 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f943c011990 con 0x7f94480c4540 2023-10-14T11:57:59.735 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.729+0000 7f94467fc700 1 -- 172.21.15.195:0/4001689632 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f943c011d20 con 0x7f94480c4540 2023-10-14T11:57:59.735 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-10-14T11:57:59.729+0000 7f94eb2a9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T11:57:59.738 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.730+0000 7f944e1f0700 1 -- 172.21.15.195:0/4001689632 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f94480c4540 msgr2=0x7f94480c02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:59.738 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.730+0000 7f944e1f0700 1 --2- 172.21.15.195:0/4001689632 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f94480c4540 0x7f94480c02e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f943c0098f0 tx=0x7f943c009c00 comp rx=0 tx=0).stop 2023-10-14T11:57:59.739 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.730+0000 7f944e1f0700 1 -- 172.21.15.195:0/4001689632 shutdown_connections 2023-10-14T11:57:59.739 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.730+0000 7f944e1f0700 1 --2- 172.21.15.195:0/4001689632 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f944806b500 0x7f94480c0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:59.739 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.730+0000 7f944e1f0700 1 --2- 172.21.15.195:0/4001689632 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f94480c4540 0x7f94480c02e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:59.739 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.730+0000 7f944e1f0700 1 -- 172.21.15.195:0/4001689632 >> 172.21.15.195:0/4001689632 conn(0x7f94480c3a50 msgr2=0x7f94480c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:57:59.739 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.730+0000 7f944e1f0700 1 -- 172.21.15.195:0/4001689632 shutdown_connections 2023-10-14T11:57:59.740 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.730+0000 7f944e1f0700 1 -- 172.21.15.195:0/4001689632 wait complete. 2023-10-14T11:57:59.740 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.731+0000 7f944e1f0700 1 Processor -- start 2023-10-14T11:57:59.740 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.731+0000 7f944e1f0700 1 -- start start 2023-10-14T11:57:59.740 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.732+0000 7f944e1f0700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f944806b500 0x7f94481384c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:59.740 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.732+0000 7f944e1f0700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f94480c4540 0x7f9448156cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:59.741 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.732+0000 7f944e1f0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f944813dbe0 0x7f9448153e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:57:59.741 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.732+0000 7f944e1f0700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9448138a00 con 0x7f944813dbe0 2023-10-14T11:57:59.741 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.732+0000 7f944e1f0700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94480c66c0 con 0x7f94480c4540 2023-10-14T11:57:59.741 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.732+0000 7f944e1f0700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94480c6910 con 0x7f944806b500 2023-10-14T11:57:59.741 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.732+0000 7f9446ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f94480c4540 0x7f9448156cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:59.742 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.732+0000 7f94477fe700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f944806b500 0x7f94481384c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:59.742 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.732+0000 7f9446ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f94480c4540 0x7f9448156cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44932/0 (socket says 172.21.15.195:44932) 2023-10-14T11:57:59.742 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.732+0000 7f94477fe700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f944806b500 0x7f94481384c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.195:39514/0 (socket says 172.21.15.195:39514) 2023-10-14T11:57:59.742 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.732+0000 7f9446ffd700 1 -- 172.21.15.195:0/811328170 learned_addr learned my addr 172.21.15.195:0/811328170 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-14T11:57:59.742 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.732+0000 7f9447fff700 1 --2- 172.21.15.195:0/811328170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f944813dbe0 0x7f9448153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:57:59.743 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.732+0000 7f9446ffd700 1 -- 172.21.15.195:0/811328170 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f944806b500 msgr2=0x7f94481384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:59.743 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.732+0000 7f9446ffd700 1 --2- 172.21.15.195:0/811328170 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f944806b500 0x7f94481384c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:59.743 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.732+0000 7f9446ffd700 1 -- 172.21.15.195:0/811328170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f944813dbe0 msgr2=0x7f9448153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:59.743 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.732+0000 7f9446ffd700 1 --2- 172.21.15.195:0/811328170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f944813dbe0 0x7f9448153e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:59.743 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.732+0000 7f9446ffd700 1 -- 172.21.15.195:0/811328170 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f943c009550 con 0x7f94480c4540 2023-10-14T11:57:59.744 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.732+0000 7f94477fe700 1 --2- 172.21.15.195:0/811328170 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f944806b500 0x7f94481384c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T11:57:59.744 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.732+0000 7f9446ffd700 1 --2- 172.21.15.195:0/811328170 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f94480c4540 0x7f9448156cf0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f943c009c40 tx=0x7f943c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:57:59.744 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.733+0000 7f942ffff700 1 -- 172.21.15.195:0/811328170 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f943c004400 con 0x7f94480c4540 2023-10-14T11:57:59.744 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.733+0000 7f942ffff700 1 -- 172.21.15.195:0/811328170 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f943c01f070 con 0x7f94480c4540 2023-10-14T11:57:59.744 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.733+0000 7f942ffff700 1 -- 172.21.15.195:0/811328170 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f943c0038e0 con 0x7f94480c4540 2023-10-14T11:57:59.744 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.733+0000 7f944e1f0700 1 -- 172.21.15.195:0/811328170 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94480c6bf0 con 0x7f94480c4540 2023-10-14T11:57:59.745 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.733+0000 7f944e1f0700 1 -- 172.21.15.195:0/811328170 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94480c5b80 con 0x7f94480c4540 2023-10-14T11:57:59.745 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.733+0000 7f942ffff700 1 -- 172.21.15.195:0/811328170 <== mon.1 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f943c021030 con 0x7f94480c4540 2023-10-14T11:57:59.745 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.733+0000 7f942ffff700 1 -- 172.21.15.195:0/811328170 <== mon.1 v2:172.21.15.195:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f943c024430 con 0x7f94480c4540 2023-10-14T11:57:59.745 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.734+0000 7f944e1f0700 1 -- 172.21.15.195:0/811328170 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9448074b80 con 0x7f94480c4540 2023-10-14T11:57:59.745 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.736+0000 7f942ffff700 1 -- 172.21.15.195:0/811328170 <== mon.1 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f943c015020 con 0x7f94480c4540 2023-10-14T11:57:59.805 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-10-14T11:57:59.801+0000 7f94eb2a9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T11:57:59.821 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.819+0000 7f944e1f0700 1 -- 172.21.15.195:0/811328170 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd new", "uuid": "47a9d4de-f56d-4aa4-9900-c4a9cf9b2f97", "id": 7} v 0) v1 -- 0x7f944806b2e0 con 0x7f94480c4540 2023-10-14T11:57:59.823 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.822+0000 7f942ffff700 1 -- 172.21.15.195:0/811328170 <== mon.1 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "47a9d4de-f56d-4aa4-9900-c4a9cf9b2f97", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f943c01a450 con 0x7f94480c4540 2023-10-14T11:57:59.824 INFO:teuthology.orchestra.run.smithi195.stdout:7 2023-10-14T11:57:59.825 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.823+0000 7f944e1f0700 1 -- 172.21.15.195:0/811328170 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f94480c4540 msgr2=0x7f9448156cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:57:59.825 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.823+0000 7f944e1f0700 1 --2- 172.21.15.195:0/811328170 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f94480c4540 0x7f9448156cf0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f943c009c40 tx=0x7f943c004750 comp rx=0 tx=0).stop 2023-10-14T11:57:59.826 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.823+0000 7f944e1f0700 1 -- 172.21.15.195:0/811328170 shutdown_connections 2023-10-14T11:57:59.826 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.823+0000 7f944e1f0700 1 --2- 172.21.15.195:0/811328170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f944813dbe0 0x7f9448153e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:59.826 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.823+0000 7f944e1f0700 1 --2- 172.21.15.195:0/811328170 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f944806b500 0x7f94481384c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:59.826 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.823+0000 7f944e1f0700 1 --2- 172.21.15.195:0/811328170 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f94480c4540 0x7f9448156cf0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:57:59.827 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.823+0000 7f944e1f0700 1 -- 172.21.15.195:0/811328170 >> 172.21.15.195:0/811328170 conn(0x7f94480c3a50 msgr2=0x7f94480c02e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:57:59.827 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.823+0000 7f944e1f0700 1 -- 172.21.15.195:0/811328170 shutdown_connections 2023-10-14T11:57:59.827 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-14T11:57:59.823+0000 7f944e1f0700 1 -- 172.21.15.195:0/811328170 wait complete. 2023-10-14T11:57:59.837 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-14T11:57:59.837 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-14T11:57:59.840 INFO:tasks.ceph.osd.0:Started 2023-10-14T11:57:59.840 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-14T11:57:59.840 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-14T11:57:59.842 INFO:tasks.ceph.osd.1:Started 2023-10-14T11:57:59.843 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-14T11:57:59.843 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-14T11:57:59.845 INFO:tasks.ceph.osd.2:Started 2023-10-14T11:57:59.846 INFO:tasks.ceph.osd.3:Restarting daemon 2023-10-14T11:57:59.846 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-10-14T11:57:59.848 INFO:tasks.ceph.osd.3:Started 2023-10-14T11:57:59.849 INFO:tasks.ceph.osd.4:Restarting daemon 2023-10-14T11:57:59.849 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-10-14T11:57:59.851 INFO:tasks.ceph.osd.4:Started 2023-10-14T11:57:59.851 INFO:tasks.ceph.osd.5:Restarting daemon 2023-10-14T11:57:59.852 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-10-14T11:57:59.854 INFO:tasks.ceph.osd.5:Started 2023-10-14T11:57:59.854 INFO:tasks.ceph.osd.6:Restarting daemon 2023-10-14T11:57:59.855 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-10-14T11:57:59.857 INFO:tasks.ceph.osd.6:Started 2023-10-14T11:57:59.857 INFO:tasks.ceph.osd.7:Restarting daemon 2023-10-14T11:57:59.858 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-10-14T11:57:59.860 INFO:tasks.ceph.osd.7:Started 2023-10-14T11:57:59.861 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T11:58:00.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.012+0000 7f040eca0700 1 Processor -- start 2023-10-14T11:58:00.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.013+0000 7f040eca0700 1 -- start start 2023-10-14T11:58:00.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.013+0000 7f040eca0700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f040806ab40 0x7f040806f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:00.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.013+0000 7f040eca0700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f04080c0350 0x7f040813d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:00.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.013+0000 7f040eca0700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f040806b630 con 0x7f04081554e0 2023-10-14T11:58:00.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.013+0000 7f040eca0700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04080c49f0 con 0x7f04080c0350 2023-10-14T11:58:00.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.013+0000 7f040eca0700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04080726a0 con 0x7f040806ab40 2023-10-14T11:58:00.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.013+0000 7f0407fff700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f04081554e0 0x7f040806af20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:48756/0 (socket says 172.21.15.78:48756) 2023-10-14T11:58:00.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.013+0000 7f040d23d700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f04080c0350 0x7f040813d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:00.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.013+0000 7f0407fff700 1 -- 172.21.15.78:0/459792341 learned_addr learned my addr 172.21.15.78:0/459792341 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:00.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.014+0000 7f040ca3c700 1 --2- 172.21.15.78:0/459792341 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f040806ab40 0x7f040806f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:00.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.014+0000 7f040d23d700 1 -- 172.21.15.78:0/459792341 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f040806ab40 msgr2=0x7f040806f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:00.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.014+0000 7f040d23d700 1 --2- 172.21.15.78:0/459792341 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f040806ab40 0x7f040806f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:00.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.014+0000 7f040d23d700 1 -- 172.21.15.78:0/459792341 >> v1:172.21.15.78:6789/0 conn(0x7f04081554e0 legacy=0x7f040806af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:00.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.014+0000 7f04077fe700 1 -- 172.21.15.78:0/459792341 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551821471 0 0) 0x7f040806b630 con 0x7f04081554e0 2023-10-14T11:58:00.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.014+0000 7f040d23d700 1 -- 172.21.15.78:0/459792341 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0408072980 con 0x7f04080c0350 2023-10-14T11:58:00.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.014+0000 7f040d23d700 1 --2- 172.21.15.78:0/459792341 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04080c0350 0x7f040813d7b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0400009d90 tx=0x7f040000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=372c32a710b2d52a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:00.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.014+0000 7f04077fe700 1 -- 172.21.15.78:0/459792341 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040000e070 con 0x7f04080c0350 2023-10-14T11:58:00.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.014+0000 7f04077fe700 1 -- 172.21.15.78:0/459792341 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f040000bb20 con 0x7f04080c0350 2023-10-14T11:58:00.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.014+0000 7f04077fe700 1 -- 172.21.15.78:0/459792341 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0400012430 con 0x7f04080c0350 2023-10-14T11:58:00.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.015+0000 7f040eca0700 1 -- 172.21.15.78:0/459792341 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04080c0350 msgr2=0x7f040813d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:00.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.015+0000 7f040eca0700 1 --2- 172.21.15.78:0/459792341 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04080c0350 0x7f040813d7b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0400009d90 tx=0x7f040000b330 comp rx=0 tx=0).stop 2023-10-14T11:58:00.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.015+0000 7f040eca0700 1 -- 172.21.15.78:0/459792341 shutdown_connections 2023-10-14T11:58:00.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.015+0000 7f040eca0700 1 --2- 172.21.15.78:0/459792341 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04080c0350 0x7f040813d7b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:00.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.015+0000 7f040eca0700 1 --2- 172.21.15.78:0/459792341 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f040806ab40 0x7f040806f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:00.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.015+0000 7f040eca0700 1 -- 172.21.15.78:0/459792341 >> 172.21.15.78:0/459792341 conn(0x7f04080c3ac0 msgr2=0x7f04080c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:00.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.015+0000 7f040eca0700 1 -- 172.21.15.78:0/459792341 shutdown_connections 2023-10-14T11:58:00.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.015+0000 7f040eca0700 1 -- 172.21.15.78:0/459792341 wait complete. 2023-10-14T11:58:00.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.016+0000 7f040eca0700 1 Processor -- start 2023-10-14T11:58:00.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.016+0000 7f040eca0700 1 -- start start 2023-10-14T11:58:00.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.017+0000 7f040eca0700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f040806ab40 0x7f04080c6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:00.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.017+0000 7f040eca0700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f04080c0350 0x7f04081339d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:00.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.017+0000 7f040eca0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04081554e0 0x7f0408133f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:00.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.017+0000 7f040eca0700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0408136c80 con 0x7f04081554e0 2023-10-14T11:58:00.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.017+0000 7f040eca0700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04081368b0 con 0x7f040806ab40 2023-10-14T11:58:00.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.017+0000 7f040eca0700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0408136010 con 0x7f04080c0350 2023-10-14T11:58:00.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.017+0000 7f040d23d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04081554e0 0x7f0408133f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:00.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.017+0000 7f040d23d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04081554e0 0x7f0408133f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43888/0 (socket says 172.21.15.78:43888) 2023-10-14T11:58:00.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.017+0000 7f040d23d700 1 -- 172.21.15.78:0/2498953047 learned_addr learned my addr 172.21.15.78:0/2498953047 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:00.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.017+0000 7f040ca3c700 1 --2- 172.21.15.78:0/2498953047 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f040806ab40 0x7f04080c6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:00.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.017+0000 7f0407fff700 1 --2- 172.21.15.78:0/2498953047 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f04080c0350 0x7f04081339d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:00.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.017+0000 7f040d23d700 1 -- 172.21.15.78:0/2498953047 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f04080c0350 msgr2=0x7f04081339d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:00.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.017+0000 7f040d23d700 1 --2- 172.21.15.78:0/2498953047 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f04080c0350 0x7f04081339d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:00.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.017+0000 7f040d23d700 1 -- 172.21.15.78:0/2498953047 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f040806ab40 msgr2=0x7f04080c6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:00.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.017+0000 7f040d23d700 1 --2- 172.21.15.78:0/2498953047 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f040806ab40 0x7f04080c6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:00.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.017+0000 7f040d23d700 1 -- 172.21.15.78:0/2498953047 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0400009a70 con 0x7f04081554e0 2023-10-14T11:58:00.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.017+0000 7f040d23d700 1 --2- 172.21.15.78:0/2498953047 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04081554e0 0x7f0408133f10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f04080ad290 tx=0x7f0400004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:00.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.017+0000 7f04057fa700 1 -- 172.21.15.78:0/2498953047 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0400012430 con 0x7f04081554e0 2023-10-14T11:58:00.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.018+0000 7f04057fa700 1 -- 172.21.15.78:0/2498953047 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0400020070 con 0x7f04081554e0 2023-10-14T11:58:00.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.018+0000 7f040eca0700 1 -- 172.21.15.78:0/2498953047 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04081362f0 con 0x7f04081554e0 2023-10-14T11:58:00.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.018+0000 7f04057fa700 1 -- 172.21.15.78:0/2498953047 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0400004280 con 0x7f04081554e0 2023-10-14T11:58:00.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.018+0000 7f040eca0700 1 -- 172.21.15.78:0/2498953047 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0408135780 con 0x7f04081554e0 2023-10-14T11:58:00.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.018+0000 7f04057fa700 1 -- 172.21.15.78:0/2498953047 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f04000046e0 con 0x7f04081554e0 2023-10-14T11:58:00.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.018+0000 7f04057fa700 1 -- 172.21.15.78:0/2498953047 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f0400022070 con 0x7f04081554e0 2023-10-14T11:58:00.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.019+0000 7f040eca0700 1 -- 172.21.15.78:0/2498953047 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0408074b80 con 0x7f04081554e0 2023-10-14T11:58:00.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.021+0000 7f04057fa700 1 -- 172.21.15.78:0/2498953047 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f0400012e20 con 0x7f04081554e0 2023-10-14T11:58:00.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.089+0000 7f040eca0700 1 -- 172.21.15.78:0/2498953047 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04080ac4d0 con 0x7f04081554e0 2023-10-14T11:58:00.092 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.090+0000 7f04057fa700 1 -- 172.21.15.78:0/2498953047 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f0400027480 con 0x7f04081554e0 2023-10-14T11:58:00.092 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:00.093 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":10,"fsid":"4c5dc134-3092-406a-a1e1-13e6d9e926e8","created":"2023-10-14T11:57:57.363750+0000","modified":"2023-10-14T11:57:59.821373+0000","last_up_change":"0.000000","last_in_change":"2023-10-14T11:57:59.821373+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"625a5fc0-9474-4eae-b80e-14c27a74ad19","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"49d63db1-2981-49ab-9c40-c50fc744b982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1d7d0b29-cbd1-4717-9fe6-765a8d4998b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2f75ded1-e9e2-40e7-99ae-45e1e425e1f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5130b432-8358-46cb-8a31-3064c976f6d8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"60e234a9-a506-4e40-9c19-49a58f3cf28c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"df7d2dd6-26b6-4eb1-96e5-94d1f3955b33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47a9d4de-f56d-4aa4-9900-c4a9cf9b2f97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T11:58:00.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.091+0000 7f040eca0700 1 -- 172.21.15.78:0/2498953047 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04081554e0 msgr2=0x7f0408133f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:00.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.091+0000 7f040eca0700 1 --2- 172.21.15.78:0/2498953047 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04081554e0 0x7f0408133f10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f04080ad290 tx=0x7f0400004ad0 comp rx=0 tx=0).stop 2023-10-14T11:58:00.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.091+0000 7f040eca0700 1 -- 172.21.15.78:0/2498953047 shutdown_connections 2023-10-14T11:58:00.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.091+0000 7f040eca0700 1 --2- 172.21.15.78:0/2498953047 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04081554e0 0x7f0408133f10 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:00.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.091+0000 7f040eca0700 1 --2- 172.21.15.78:0/2498953047 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f040806ab40 0x7f04080c6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:00.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.091+0000 7f040eca0700 1 --2- 172.21.15.78:0/2498953047 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f04080c0350 0x7f04081339d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:00.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.091+0000 7f040eca0700 1 -- 172.21.15.78:0/2498953047 >> 172.21.15.78:0/2498953047 conn(0x7f04080c3ac0 msgr2=0x7f04080667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:00.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.091+0000 7f040eca0700 1 -- 172.21.15.78:0/2498953047 shutdown_connections 2023-10-14T11:58:00.096 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:00.091+0000 7f040eca0700 1 -- 172.21.15.78:0/2498953047 wait complete. 2023-10-14T11:58:00.105 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-14T11:58:00.105 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-14T11:58:00.221 INFO:tasks.ceph.osd.0.smithi078.stderr:2023-10-14T11:58:00.219+0000 7ff1f8325640 -1 Falling back to public interface 2023-10-14T11:58:00.447 INFO:tasks.ceph.mgr.y.smithi078.stderr:2023-10-14T11:58:00.445+0000 7ff4d96d6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T11:58:00.453 INFO:tasks.ceph.osd.1.smithi078.stderr:2023-10-14T11:58:00.451+0000 7fefe36fa640 -1 Falling back to public interface 2023-10-14T11:58:00.467 INFO:tasks.ceph.osd.4.smithi195.stderr:2023-10-14T11:58:00.465+0000 7f6ca2d61640 -1 Falling back to public interface 2023-10-14T11:58:00.468 INFO:tasks.ceph.osd.3.smithi078.stderr:2023-10-14T11:58:00.465+0000 7f37d9223640 -1 Falling back to public interface 2023-10-14T11:58:00.473 INFO:tasks.ceph.osd.2.smithi078.stderr:2023-10-14T11:58:00.471+0000 7f131324d640 -1 Falling back to public interface 2023-10-14T11:58:00.476 INFO:tasks.ceph.osd.6.smithi195.stderr:2023-10-14T11:58:00.474+0000 7f623c01b640 -1 Falling back to public interface 2023-10-14T11:58:00.476 INFO:tasks.ceph.osd.7.smithi195.stderr:2023-10-14T11:58:00.474+0000 7f80e2a48640 -1 Falling back to public interface 2023-10-14T11:58:00.484 INFO:tasks.ceph.osd.5.smithi195.stderr:2023-10-14T11:58:00.482+0000 7fa73eb48640 -1 Falling back to public interface 2023-10-14T11:58:00.526 INFO:tasks.ceph.mgr.y.smithi078.stderr:2023-10-14T11:58:00.524+0000 7ff4d96d6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T11:58:00.542 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-10-14T11:58:00.540+0000 7f94eb2a9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T11:58:00.618 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-10-14T11:58:00.617+0000 7f94eb2a9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T11:58:00.666 INFO:tasks.ceph.mgr.y.smithi078.stderr:2023-10-14T11:58:00.664+0000 7ff4d96d6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T11:58:00.761 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-10-14T11:58:00.759+0000 7f94eb2a9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T11:58:00.901 INFO:tasks.ceph.mgr.y.smithi078.stderr:2023-10-14T11:58:00.899+0000 7ff4d96d6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T11:58:00.911 DEBUG:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-14T11:58:00.993 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-10-14T11:58:00.992+0000 7f94eb2a9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T11:58:01.026 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.024+0000 7fc05345e700 1 Processor -- start 2023-10-14T11:58:01.027 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.025+0000 7fc05345e700 1 -- start start 2023-10-14T11:58:01.028 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.026+0000 7fc05345e700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc04c0c4b80 0x7fc04c0c4f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:01.029 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.026+0000 7fc05345e700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc04c06c4b0 con 0x7fc04c06b3a0 2023-10-14T11:58:01.029 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.026+0000 7fc05345e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc04c0c56d0 con 0x7fc04c1549e0 2023-10-14T11:58:01.029 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.026+0000 7fc05345e700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc04c06b8a0 con 0x7fc04c0c4b80 2023-10-14T11:58:01.029 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.026+0000 7fc0511fa700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc04c0c4b80 0x7fc04c0c4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:01.029 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.026+0000 7fc0511fa700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc04c0c4b80 0x7fc04c0c4f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43890/0 (socket says 172.21.15.78:43890) 2023-10-14T11:58:01.030 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.026+0000 7fc0509f9700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc04c06b3a0 0x7fc04c06c940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:48766/0 (socket says 172.21.15.78:48766) 2023-10-14T11:58:01.030 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.026+0000 7fc0511fa700 1 -- 172.21.15.78:0/863258781 learned_addr learned my addr 172.21.15.78:0/863258781 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:01.030 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.026+0000 7fc0511fa700 1 -- 172.21.15.78:0/863258781 >> v1:172.21.15.195:6789/0 conn(0x7fc04c1549e0 legacy=0x7fc04c0c8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:01.030 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.026+0000 7fc0511fa700 1 -- 172.21.15.78:0/863258781 >> v1:172.21.15.78:6789/0 conn(0x7fc04c06b3a0 legacy=0x7fc04c06c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:01.031 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.026+0000 7fc0511fa700 1 -- 172.21.15.78:0/863258781 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc04c06bb80 con 0x7fc04c0c4b80 2023-10-14T11:58:01.031 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.026+0000 7fc0511fa700 1 --2- 172.21.15.78:0/863258781 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc04c0c4b80 0x7fc04c0c4f60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc0480098f0 tx=0x7fc048009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=643d697366b8034e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:01.031 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.026+0000 7fc043fff700 1 -- 172.21.15.78:0/863258781 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc04800b8a0 con 0x7fc04c0c4b80 2023-10-14T11:58:01.031 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.026+0000 7fc043fff700 1 -- 172.21.15.78:0/863258781 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc04800ba00 con 0x7fc04c0c4b80 2023-10-14T11:58:01.031 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.026+0000 7fc043fff700 1 -- 172.21.15.78:0/863258781 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc04800bd90 con 0x7fc04c0c4b80 2023-10-14T11:58:01.034 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.027+0000 7fc05345e700 1 -- 172.21.15.78:0/863258781 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc04c0c4b80 msgr2=0x7fc04c0c4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:01.035 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.027+0000 7fc05345e700 1 --2- 172.21.15.78:0/863258781 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc04c0c4b80 0x7fc04c0c4f60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc0480098f0 tx=0x7fc048009c00 comp rx=0 tx=0).stop 2023-10-14T11:58:01.035 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.027+0000 7fc05345e700 1 -- 172.21.15.78:0/863258781 shutdown_connections 2023-10-14T11:58:01.035 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.027+0000 7fc05345e700 1 --2- 172.21.15.78:0/863258781 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc04c0c4b80 0x7fc04c0c4f60 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:01.035 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.027+0000 7fc05345e700 1 -- 172.21.15.78:0/863258781 >> 172.21.15.78:0/863258781 conn(0x7fc04c0ca200 msgr2=0x7fc04c0ca5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:01.036 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.027+0000 7fc05345e700 1 -- 172.21.15.78:0/863258781 shutdown_connections 2023-10-14T11:58:01.036 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.027+0000 7fc05345e700 1 -- 172.21.15.78:0/863258781 wait complete. 2023-10-14T11:58:01.036 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.029+0000 7fc05345e700 1 Processor -- start 2023-10-14T11:58:01.036 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.029+0000 7fc05345e700 1 -- start start 2023-10-14T11:58:01.037 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.030+0000 7fc05345e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc04c06b3a0 0x7fc04c0c12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:01.037 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.030+0000 7fc05345e700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc04c0c4b80 0x7fc04c1eabb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:01.037 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.030+0000 7fc05345e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc04c1549e0 0x7fc04c1508d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:01.037 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.030+0000 7fc05345e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c145350 con 0x7fc04c06b3a0 2023-10-14T11:58:01.037 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.030+0000 7fc05345e700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c1eb0f0 con 0x7fc04c1549e0 2023-10-14T11:58:01.038 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.030+0000 7fc05345e700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c150340 con 0x7fc04c0c4b80 2023-10-14T11:58:01.038 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.030+0000 7fc0511fa700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc04c06b3a0 0x7fc04c0c12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:01.038 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.030+0000 7fc0511fa700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc04c06b3a0 0x7fc04c0c12d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43896/0 (socket says 172.21.15.78:43896) 2023-10-14T11:58:01.038 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.030+0000 7fc0511fa700 1 -- 172.21.15.78:0/3959929761 learned_addr learned my addr 172.21.15.78:0/3959929761 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:01.042 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.030+0000 7fc0509f9700 1 --2- 172.21.15.78:0/3959929761 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc04c0c4b80 0x7fc04c1eabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:01.042 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.030+0000 7fc0519fb700 1 --2- 172.21.15.78:0/3959929761 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc04c1549e0 0x7fc04c1508d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:01.042 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.030+0000 7fc0511fa700 1 -- 172.21.15.78:0/3959929761 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc04c0c4b80 msgr2=0x7fc04c1eabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:01.042 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.030+0000 7fc0511fa700 1 --2- 172.21.15.78:0/3959929761 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc04c0c4b80 0x7fc04c1eabb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:01.043 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.030+0000 7fc0511fa700 1 -- 172.21.15.78:0/3959929761 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc04c1549e0 msgr2=0x7fc04c1508d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:01.043 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.030+0000 7fc0511fa700 1 --2- 172.21.15.78:0/3959929761 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc04c1549e0 0x7fc04c1508d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:01.043 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.030+0000 7fc0511fa700 1 -- 172.21.15.78:0/3959929761 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc048009550 con 0x7fc04c06b3a0 2023-10-14T11:58:01.043 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.031+0000 7fc0511fa700 1 --2- 172.21.15.78:0/3959929761 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc04c06b3a0 0x7fc04c0c12d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc048000c00 tx=0x7fc048004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:01.043 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.031+0000 7fc041ffb700 1 -- 172.21.15.78:0/3959929761 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc048004380 con 0x7fc04c06b3a0 2023-10-14T11:58:01.044 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.031+0000 7fc041ffb700 1 -- 172.21.15.78:0/3959929761 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc04801f070 con 0x7fc04c06b3a0 2023-10-14T11:58:01.044 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.031+0000 7fc05345e700 1 -- 172.21.15.78:0/3959929761 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc04c150e10 con 0x7fc04c06b3a0 2023-10-14T11:58:01.044 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.031+0000 7fc041ffb700 1 -- 172.21.15.78:0/3959929761 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0480036a0 con 0x7fc04c06b3a0 2023-10-14T11:58:01.044 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.031+0000 7fc05345e700 1 -- 172.21.15.78:0/3959929761 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc04c14f5e0 con 0x7fc04c06b3a0 2023-10-14T11:58:01.045 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.031+0000 7fc041ffb700 1 -- 172.21.15.78:0/3959929761 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc048021030 con 0x7fc04c06b3a0 2023-10-14T11:58:01.045 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.031+0000 7fc041ffb700 1 -- 172.21.15.78:0/3959929761 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fc04802d440 con 0x7fc04c06b3a0 2023-10-14T11:58:01.045 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.032+0000 7fc05345e700 1 -- 172.21.15.78:0/3959929761 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc04c04f400 con 0x7fc04c06b3a0 2023-10-14T11:58:01.046 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.034+0000 7fc041ffb700 1 -- 172.21.15.78:0/3959929761 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fc048015020 con 0x7fc04c06b3a0 2023-10-14T11:58:01.106 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.104+0000 7fc05345e700 1 -- 172.21.15.78:0/3959929761 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc04c0c8100 con 0x7fc04c06b3a0 2023-10-14T11:58:01.107 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.105+0000 7fc041ffb700 1 -- 172.21.15.78:0/3959929761 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fc048003ce0 con 0x7fc04c06b3a0 2023-10-14T11:58:01.107 INFO:teuthology.misc.health.smithi078.stdout: 2023-10-14T11:58:01.107 INFO:teuthology.misc.health.smithi078.stdout:{"epoch":10,"fsid":"4c5dc134-3092-406a-a1e1-13e6d9e926e8","created":"2023-10-14T11:57:57.363750+0000","modified":"2023-10-14T11:57:59.821373+0000","last_up_change":"0.000000","last_in_change":"2023-10-14T11:57:59.821373+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"625a5fc0-9474-4eae-b80e-14c27a74ad19","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"49d63db1-2981-49ab-9c40-c50fc744b982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1d7d0b29-cbd1-4717-9fe6-765a8d4998b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2f75ded1-e9e2-40e7-99ae-45e1e425e1f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5130b432-8358-46cb-8a31-3064c976f6d8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"60e234a9-a506-4e40-9c19-49a58f3cf28c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"df7d2dd6-26b6-4eb1-96e5-94d1f3955b33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47a9d4de-f56d-4aa4-9900-c4a9cf9b2f97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T11:58:01.108 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.106+0000 7fc05345e700 1 -- 172.21.15.78:0/3959929761 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc04c06b3a0 msgr2=0x7fc04c0c12d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:01.109 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.106+0000 7fc05345e700 1 --2- 172.21.15.78:0/3959929761 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc04c06b3a0 0x7fc04c0c12d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc048000c00 tx=0x7fc048004750 comp rx=0 tx=0).stop 2023-10-14T11:58:01.109 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.106+0000 7fc05345e700 1 -- 172.21.15.78:0/3959929761 shutdown_connections 2023-10-14T11:58:01.109 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.106+0000 7fc05345e700 1 --2- 172.21.15.78:0/3959929761 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc04c1549e0 0x7fc04c1508d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:01.109 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.106+0000 7fc05345e700 1 --2- 172.21.15.78:0/3959929761 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc04c06b3a0 0x7fc04c0c12d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:01.109 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.106+0000 7fc05345e700 1 --2- 172.21.15.78:0/3959929761 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc04c0c4b80 0x7fc04c1eabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:01.110 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.106+0000 7fc05345e700 1 -- 172.21.15.78:0/3959929761 >> 172.21.15.78:0/3959929761 conn(0x7fc04c0ca200 msgr2=0x7fc04c0ab600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:01.110 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.106+0000 7fc05345e700 1 -- 172.21.15.78:0/3959929761 shutdown_connections 2023-10-14T11:58:01.110 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:01.106+0000 7fc05345e700 1 -- 172.21.15.78:0/3959929761 wait complete. 2023-10-14T11:58:01.118 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-14T11:58:01.122 INFO:tasks.ceph.mgr.y.smithi078.stderr:2023-10-14T11:58:01.120+0000 7ff4d96d6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T11:58:01.208 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-10-14T11:58:01.206+0000 7f94eb2a9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T11:58:01.255 INFO:tasks.ceph.mgr.y.smithi078.stderr:2023-10-14T11:58:01.253+0000 7ff4d96d6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T11:58:01.342 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-10-14T11:58:01.340+0000 7f94eb2a9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T11:58:01.473 INFO:tasks.ceph.mgr.y.smithi078.stderr:2023-10-14T11:58:01.471+0000 7ff4d96d6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T11:58:01.560 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-10-14T11:58:01.558+0000 7f94eb2a9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T11:58:01.630 INFO:tasks.ceph.mgr.y.smithi078.stderr:2023-10-14T11:58:01.627+0000 7ff4d96d6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T11:58:01.716 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-10-14T11:58:01.715+0000 7f94eb2a9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T11:58:01.742 INFO:tasks.ceph.mgr.y.smithi078.stderr:2023-10-14T11:58:01.740+0000 7ff4d96d6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T11:58:01.830 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-10-14T11:58:01.829+0000 7f94eb2a9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T11:58:02.062 INFO:tasks.ceph.mgr.y.smithi078.stderr:2023-10-14T11:58:02.060+0000 7ff4d96d6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T11:58:02.156 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-10-14T11:58:02.155+0000 7f94eb2a9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T11:58:02.162 INFO:tasks.ceph.mgr.y.smithi078.stderr:2023-10-14T11:58:02.160+0000 7ff4d96d6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T11:58:02.256 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-10-14T11:58:02.254+0000 7f94eb2a9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T11:58:02.346 INFO:tasks.ceph.mgr.y.smithi078.stderr:2023-10-14T11:58:02.344+0000 7ff4d96d6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T11:58:02.441 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-10-14T11:58:02.439+0000 7f94eb2a9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T11:58:02.938 INFO:tasks.ceph.mgr.y.smithi078.stderr:2023-10-14T11:58:02.936+0000 7ff4d96d6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T11:58:03.041 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-10-14T11:58:03.040+0000 7f94eb2a9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T11:58:03.749 INFO:tasks.ceph.osd.0.smithi078.stderr:2023-10-14T11:58:03.748+0000 7ff1f8325640 -1 osd.0 0 log_to_monitors true 2023-10-14T11:58:03.961 INFO:tasks.ceph.mgr.y.smithi078.stderr:2023-10-14T11:58:03.959+0000 7ff491664700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-10-14T11:58:03.961 INFO:tasks.ceph.mgr.y.smithi078.stderr:2023-10-14T11:58:03.959+0000 7ff491664700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-10-14T11:58:03.992 INFO:tasks.ceph.osd.1.smithi078.stderr:2023-10-14T11:58:03.990+0000 7fefe36fa640 -1 osd.1 0 log_to_monitors true 2023-10-14T11:58:04.021 INFO:tasks.ceph.osd.6.smithi195.stderr:2023-10-14T11:58:04.019+0000 7f623c01b640 -1 osd.6 0 log_to_monitors true 2023-10-14T11:58:04.021 INFO:tasks.ceph.osd.4.smithi195.stderr:2023-10-14T11:58:04.019+0000 7f6ca2d61640 -1 osd.4 0 log_to_monitors true 2023-10-14T11:58:04.022 INFO:tasks.ceph.osd.3.smithi078.stderr:2023-10-14T11:58:04.020+0000 7f37d9223640 -1 osd.3 0 log_to_monitors true 2023-10-14T11:58:04.025 INFO:tasks.ceph.osd.2.smithi078.stderr:2023-10-14T11:58:04.024+0000 7f131324d640 -1 osd.2 0 log_to_monitors true 2023-10-14T11:58:04.037 INFO:tasks.ceph.osd.7.smithi195.stderr:2023-10-14T11:58:04.035+0000 7f80e2a48640 -1 osd.7 0 log_to_monitors true 2023-10-14T11:58:04.038 INFO:tasks.ceph.osd.5.smithi195.stderr:2023-10-14T11:58:04.037+0000 7fa73eb48640 -1 osd.5 0 log_to_monitors true 2023-10-14T11:58:05.979 INFO:tasks.ceph.osd.3.smithi078.stderr:2023-10-14T11:58:05.977+0000 7f37cd5db700 -1 osd.3 0 waiting for initial osdmap 2023-10-14T11:58:05.994 INFO:tasks.ceph.osd.3.smithi078.stderr:2023-10-14T11:58:05.992+0000 7f37c51b5700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-14T11:58:06.050 INFO:tasks.ceph.osd.1.smithi078.stderr:2023-10-14T11:58:06.047+0000 7fefd7ab2700 -1 osd.1 0 waiting for initial osdmap 2023-10-14T11:58:06.059 INFO:tasks.ceph.osd.1.smithi078.stderr:2023-10-14T11:58:06.058+0000 7fefcf68c700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-14T11:58:06.064 INFO:tasks.ceph.osd.6.smithi195.stderr:2023-10-14T11:58:06.060+0000 7f62303d3700 -1 osd.6 0 waiting for initial osdmap 2023-10-14T11:58:06.066 INFO:tasks.ceph.osd.7.smithi195.stderr:2023-10-14T11:58:06.063+0000 7f80d6e00700 -1 osd.7 0 waiting for initial osdmap 2023-10-14T11:58:06.074 INFO:tasks.ceph.osd.5.smithi195.stderr:2023-10-14T11:58:06.073+0000 7fa732f00700 -1 osd.5 0 waiting for initial osdmap 2023-10-14T11:58:06.077 INFO:tasks.ceph.osd.7.smithi195.stderr:2023-10-14T11:58:06.076+0000 7f80ce9da700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-14T11:58:06.078 INFO:tasks.ceph.osd.6.smithi195.stderr:2023-10-14T11:58:06.077+0000 7f6227fad700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-14T11:58:06.086 INFO:tasks.ceph.osd.5.smithi195.stderr:2023-10-14T11:58:06.085+0000 7fa72aada700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-14T11:58:06.087 INFO:tasks.ceph.osd.0.smithi078.stderr:2023-10-14T11:58:06.085+0000 7ff1ec6dd700 -1 osd.0 0 waiting for initial osdmap 2023-10-14T11:58:06.099 INFO:tasks.ceph.osd.0.smithi078.stderr:2023-10-14T11:58:06.097+0000 7ff1e42b7700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-14T11:58:06.143 INFO:tasks.ceph.osd.4.smithi195.stderr:2023-10-14T11:58:06.141+0000 7f6c97119700 -1 osd.4 0 waiting for initial osdmap 2023-10-14T11:58:06.154 INFO:tasks.ceph.osd.4.smithi195.stderr:2023-10-14T11:58:06.153+0000 7f6c8ecf3700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-14T11:58:06.297 INFO:tasks.ceph.osd.2.smithi078.stderr:2023-10-14T11:58:06.295+0000 7f1307605700 -1 osd.2 0 waiting for initial osdmap 2023-10-14T11:58:06.306 INFO:tasks.ceph.osd.2.smithi078.stderr:2023-10-14T11:58:06.305+0000 7f12ff1df700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-14T11:58:07.925 DEBUG:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-14T11:58:08.052 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.050+0000 7f90ba250700 1 Processor -- start 2023-10-14T11:58:08.053 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.050+0000 7f90ba250700 1 -- start start 2023-10-14T11:58:08.060 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.051+0000 7f90ba250700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f90b406fa90 0x7f90b4072220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:08.060 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.051+0000 7f90ba250700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b406ada0 con 0x7f90b4071e40 2023-10-14T11:58:08.061 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.051+0000 7f90ba250700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b4153560 con 0x7f90b406f660 2023-10-14T11:58:08.061 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.051+0000 7f90ba250700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90b406b420 con 0x7f90b406fa90 2023-10-14T11:58:08.061 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.051+0000 7f90b37fe700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f90b4071e40 0x7f90b4155ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49012/0 (socket says 172.21.15.78:49012) 2023-10-14T11:58:08.061 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.051+0000 7f90b37fe700 1 -- 172.21.15.78:0/2846781532 learned_addr learned my addr 172.21.15.78:0/2846781532 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:08.062 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.051+0000 7f90b3fff700 1 --2- 172.21.15.78:0/2846781532 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f90b406fa90 0x7f90b4072220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:08.062 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.051+0000 7f90b27fc700 1 -- 172.21.15.78:0/2846781532 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1170630473 0 0) 0x7f90b406ada0 con 0x7f90b4071e40 2023-10-14T11:58:08.062 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.051+0000 7f90b27fc700 1 -- 172.21.15.78:0/2846781532 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90a00035b0 con 0x7f90b4071e40 2023-10-14T11:58:08.062 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.051+0000 7f90b27fc700 1 -- 172.21.15.78:0/2846781532 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 740925635 0 0) 0x7f90a00035b0 con 0x7f90b4071e40 2023-10-14T11:58:08.062 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.051+0000 7f90b27fc700 1 -- 172.21.15.78:0/2846781532 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f90b406fa90 msgr2=0x7f90b4072220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:08.063 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.051+0000 7f90b27fc700 1 --2- 172.21.15.78:0/2846781532 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f90b406fa90 0x7f90b4072220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:08.063 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.051+0000 7f90b27fc700 1 -- 172.21.15.78:0/2846781532 >> v1:172.21.15.195:6789/0 conn(0x7f90b406f660 legacy=0x7f90b41565b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:08.063 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.051+0000 7f90b27fc700 1 -- 172.21.15.78:0/2846781532 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b41531f0 con 0x7f90b4071e40 2023-10-14T11:58:08.063 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.051+0000 7f90b27fc700 1 -- 172.21.15.78:0/2846781532 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441799917 0 0) 0x7f90a4001670 con 0x7f90b4071e40 2023-10-14T11:58:08.064 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.052+0000 7f90b27fc700 1 -- 172.21.15.78:0/2846781532 >> v1:172.21.15.78:6789/0 conn(0x7f90b4071e40 legacy=0x7f90b4155ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:08.064 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.052+0000 7f90b27fc700 1 --2- 172.21.15.78:0/2846781532 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f90b406fa90 0x7f90a0003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:08.064 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.052+0000 7f90b27fc700 1 --2- 172.21.15.78:0/2846781532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90a0003fe0 0x7f90a0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:08.064 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.052+0000 7f90b27fc700 1 --2- 172.21.15.78:0/2846781532 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f90a0004d90 0x7f90a0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:08.064 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.052+0000 7f90b27fc700 1 -- 172.21.15.78:0/2846781532 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b406b420 con 0x7f90a0003fe0 2023-10-14T11:58:08.064 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.052+0000 7f90b27fc700 1 -- 172.21.15.78:0/2846781532 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a000bc20 con 0x7f90a0004d90 2023-10-14T11:58:08.065 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.052+0000 7f90b27fc700 1 -- 172.21.15.78:0/2846781532 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90a000bde0 con 0x7f90b406fa90 2023-10-14T11:58:08.065 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.052+0000 7f90b2ffd700 1 --2- 172.21.15.78:0/2846781532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90a0003fe0 0x7f90a0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:08.065 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.052+0000 7f90b37fe700 1 --2- 172.21.15.78:0/2846781532 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f90b406fa90 0x7f90a0003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:08.065 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.052+0000 7f90b3fff700 1 --2- 172.21.15.78:0/2846781532 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f90a0004d90 0x7f90a0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:08.065 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.052+0000 7f90b2ffd700 1 -- 172.21.15.78:0/2846781532 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f90b406fa90 msgr2=0x7f90a0003aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:08.066 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.052+0000 7f90b2ffd700 1 --2- 172.21.15.78:0/2846781532 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f90b406fa90 0x7f90a0003aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:08.066 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.052+0000 7f90b2ffd700 1 -- 172.21.15.78:0/2846781532 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f90a0004d90 msgr2=0x7f90a0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:08.066 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.052+0000 7f90b2ffd700 1 --2- 172.21.15.78:0/2846781532 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f90a0004d90 0x7f90a0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:08.066 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.052+0000 7f90b2ffd700 1 -- 172.21.15.78:0/2846781532 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a00035b0 con 0x7f90a0003fe0 2023-10-14T11:58:08.066 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.052+0000 7f90b2ffd700 1 --2- 172.21.15.78:0/2846781532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90a0003fe0 0x7f90a0004850 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f909c0018d0 tx=0x7f90a0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b611199e61104b43 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:08.067 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.052+0000 7f90b27fc700 1 -- 172.21.15.78:0/2846781532 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909c00c690 con 0x7f90a0003fe0 2023-10-14T11:58:08.067 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.052+0000 7f90b27fc700 1 -- 172.21.15.78:0/2846781532 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f909c00c7f0 con 0x7f90a0003fe0 2023-10-14T11:58:08.067 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.052+0000 7f90b27fc700 1 -- 172.21.15.78:0/2846781532 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909c00cba0 con 0x7f90a0003fe0 2023-10-14T11:58:08.067 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.053+0000 7f90ba250700 1 -- 172.21.15.78:0/2846781532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90a0003fe0 msgr2=0x7f90a0004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:08.067 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.053+0000 7f90ba250700 1 --2- 172.21.15.78:0/2846781532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90a0003fe0 0x7f90a0004850 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f909c0018d0 tx=0x7f90a0005880 comp rx=0 tx=0).stop 2023-10-14T11:58:08.068 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.053+0000 7f90b3fff700 1 -- 172.21.15.78:0/2846781532 reap_dead start 2023-10-14T11:58:08.068 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.053+0000 7f90ba250700 1 -- 172.21.15.78:0/2846781532 shutdown_connections 2023-10-14T11:58:08.068 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.053+0000 7f90ba250700 1 -- 172.21.15.78:0/2846781532 >> 172.21.15.78:0/2846781532 conn(0x7f90b4071320 msgr2=0x7f90b4071730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:08.068 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.053+0000 7f90ba250700 1 -- 172.21.15.78:0/2846781532 shutdown_connections 2023-10-14T11:58:08.068 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.053+0000 7f90ba250700 1 -- 172.21.15.78:0/2846781532 wait complete. 2023-10-14T11:58:08.069 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.054+0000 7f90ba250700 1 Processor -- start 2023-10-14T11:58:08.069 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.054+0000 7f90ba250700 1 -- start start 2023-10-14T11:58:08.069 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.055+0000 7f90ba250700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f90a0003fe0 0x7f90b40ac350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:08.069 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.055+0000 7f90ba250700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f90b40c35b0 0x7f90b413c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:08.069 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.055+0000 7f90ba250700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90b4149a60 0x7f90b413cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:08.069 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.055+0000 7f90ba250700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b413d120 con 0x7f90b4149a60 2023-10-14T11:58:08.070 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.055+0000 7f90ba250700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b4150440 con 0x7f90b40c35b0 2023-10-14T11:58:08.070 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.055+0000 7f90ba250700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90b4150720 con 0x7f90a0003fe0 2023-10-14T11:58:08.070 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.055+0000 7f90b3fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90b4149a60 0x7f90b413cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:08.070 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.055+0000 7f90b3fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90b4149a60 0x7f90b413cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44152/0 (socket says 172.21.15.78:44152) 2023-10-14T11:58:08.070 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.055+0000 7f90b3fff700 1 -- 172.21.15.78:0/798113300 learned_addr learned my addr 172.21.15.78:0/798113300 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:08.071 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.055+0000 7f90b2ffd700 1 --2- 172.21.15.78:0/798113300 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f90b40c35b0 0x7f90b413c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:08.071 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.055+0000 7f90b37fe700 1 --2- 172.21.15.78:0/798113300 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f90a0003fe0 0x7f90b40ac350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:08.071 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.055+0000 7f90b3fff700 1 -- 172.21.15.78:0/798113300 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f90a0003fe0 msgr2=0x7f90b40ac350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:08.071 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.055+0000 7f90b3fff700 1 --2- 172.21.15.78:0/798113300 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f90a0003fe0 0x7f90b40ac350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:08.071 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.055+0000 7f90b3fff700 1 -- 172.21.15.78:0/798113300 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f90b40c35b0 msgr2=0x7f90b413c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:08.072 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.055+0000 7f90b3fff700 1 --2- 172.21.15.78:0/798113300 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f90b40c35b0 0x7f90b413c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:08.072 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.055+0000 7f90b3fff700 1 -- 172.21.15.78:0/798113300 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b4150a00 con 0x7f90b4149a60 2023-10-14T11:58:08.072 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.055+0000 7f90b3fff700 1 --2- 172.21.15.78:0/798113300 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90b4149a60 0x7f90b413cbe0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f90b406fea0 tx=0x7f90ac011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:08.072 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.056+0000 7f90abfff700 1 -- 172.21.15.78:0/798113300 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac00e040 con 0x7f90b4149a60 2023-10-14T11:58:08.072 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.056+0000 7f90abfff700 1 -- 172.21.15.78:0/798113300 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90ac00f040 con 0x7f90b4149a60 2023-10-14T11:58:08.073 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.056+0000 7f90abfff700 1 -- 172.21.15.78:0/798113300 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac00f870 con 0x7f90b4149a60 2023-10-14T11:58:08.073 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.056+0000 7f90ba250700 1 -- 172.21.15.78:0/798113300 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90b41537f0 con 0x7f90b4149a60 2023-10-14T11:58:08.073 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.056+0000 7f90ba250700 1 -- 172.21.15.78:0/798113300 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90b40ac890 con 0x7f90b4149a60 2023-10-14T11:58:08.073 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.057+0000 7f90ba250700 1 -- 172.21.15.78:0/798113300 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90b404f400 con 0x7f90b4149a60 2023-10-14T11:58:08.074 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.057+0000 7f90abfff700 1 -- 172.21.15.78:0/798113300 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 4) v1 ==== 89173+0+0 (secure 0 0 0) 0x7f90ac00f9d0 con 0x7f90b4149a60 2023-10-14T11:58:08.074 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.057+0000 7f90abfff700 1 --2- 172.21.15.78:0/798113300 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f909406ae70 0x7f909406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:08.074 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.057+0000 7f90abfff700 1 -- 172.21.15.78:0/798113300 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f90ac089540 con 0x7f90b4149a60 2023-10-14T11:58:08.074 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.057+0000 7f90b37fe700 1 --2- 172.21.15.78:0/798113300 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f909406ae70 0x7f909406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:08.074 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.058+0000 7f90b37fe700 1 --2- 172.21.15.78:0/798113300 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f909406ae70 0x7f909406d330 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f90a4001eb0 tx=0x7f90a4000f80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:08.075 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.062+0000 7f90abfff700 1 -- 172.21.15.78:0/798113300 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f90ac054bb0 con 0x7f90b4149a60 2023-10-14T11:58:08.256 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.254+0000 7f90ba250700 1 -- 172.21.15.78:0/798113300 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90b40a8910 con 0x7f90b4149a60 2023-10-14T11:58:08.256 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.255+0000 7f90abfff700 1 -- 172.21.15.78:0/798113300 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+11022 (secure 0 0 0) 0x7f90ac0581d0 con 0x7f90b4149a60 2023-10-14T11:58:08.257 INFO:teuthology.misc.health.smithi078.stdout: 2023-10-14T11:58:08.257 INFO:teuthology.misc.health.smithi078.stdout:{"epoch":14,"fsid":"4c5dc134-3092-406a-a1e1-13e6d9e926e8","created":"2023-10-14T11:57:57.363750+0000","modified":"2023-10-14T11:58:07.984311+0000","last_up_change":"2023-10-14T11:58:06.982472+0000","last_in_change":"2023-10-14T11:57:59.821373+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"625a5fc0-9474-4eae-b80e-14c27a74ad19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6801","nonce":3884969780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6803","nonce":3884969780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6807","nonce":3884969780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6805","nonce":3884969780}]},"public_addr":"172.21.15.78:6801/3884969780","cluster_addr":"172.21.15.78:6803/3884969780","heartbeat_back_addr":"172.21.15.78:6807/3884969780","heartbeat_front_addr":"172.21.15.78:6805/3884969780","state":["exists","up"]},{"osd":1,"uuid":"49d63db1-2981-49ab-9c40-c50fc744b982","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6809","nonce":2176928406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6811","nonce":2176928406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6815","nonce":2176928406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6813","nonce":2176928406}]},"public_addr":"172.21.15.78:6809/2176928406","cluster_addr":"172.21.15.78:6811/2176928406","heartbeat_back_addr":"172.21.15.78:6815/2176928406","heartbeat_front_addr":"172.21.15.78:6813/2176928406","state":["exists","up"]},{"osd":2,"uuid":"1d7d0b29-cbd1-4717-9fe6-765a8d4998b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6825","nonce":3647044577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6827","nonce":3647044577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6831","nonce":3647044577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6829","nonce":3647044577}]},"public_addr":"172.21.15.78:6825/3647044577","cluster_addr":"172.21.15.78:6827/3647044577","heartbeat_back_addr":"172.21.15.78:6831/3647044577","heartbeat_front_addr":"172.21.15.78:6829/3647044577","state":["exists","up"]},{"osd":3,"uuid":"2f75ded1-e9e2-40e7-99ae-45e1e425e1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6817","nonce":3584112288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6819","nonce":3584112288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6823","nonce":3584112288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6821","nonce":3584112288}]},"public_addr":"172.21.15.78:6817/3584112288","cluster_addr":"172.21.15.78:6819/3584112288","heartbeat_back_addr":"172.21.15.78:6823/3584112288","heartbeat_front_addr":"172.21.15.78:6821/3584112288","state":["exists","up"]},{"osd":4,"uuid":"5130b432-8358-46cb-8a31-3064c976f6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6801","nonce":3732756301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6803","nonce":3732756301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6807","nonce":3732756301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6805","nonce":3732756301}]},"public_addr":"172.21.15.195:6801/3732756301","cluster_addr":"172.21.15.195:6803/3732756301","heartbeat_back_addr":"172.21.15.195:6807/3732756301","heartbeat_front_addr":"172.21.15.195:6805/3732756301","state":["exists","up"]},{"osd":5,"uuid":"60e234a9-a506-4e40-9c19-49a58f3cf28c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6825","nonce":2857706411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6827","nonce":2857706411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6831","nonce":2857706411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6829","nonce":2857706411}]},"public_addr":"172.21.15.195:6825/2857706411","cluster_addr":"172.21.15.195:6827/2857706411","heartbeat_back_addr":"172.21.15.195:6831/2857706411","heartbeat_front_addr":"172.21.15.195:6829/2857706411","state":["exists","up"]},{"osd":6,"uuid":"df7d2dd6-26b6-4eb1-96e5-94d1f3955b33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6810","nonce":72554374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6813","nonce":72554374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6820","nonce":72554374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6817","nonce":72554374}]},"public_addr":"172.21.15.195:6810/72554374","cluster_addr":"172.21.15.195:6813/72554374","heartbeat_back_addr":"172.21.15.195:6820/72554374","heartbeat_front_addr":"172.21.15.195:6817/72554374","state":["exists","up"]},{"osd":7,"uuid":"47a9d4de-f56d-4aa4-9900-c4a9cf9b2f97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6809","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6811","nonce":482472979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6816","nonce":482472979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6823","nonce":482472979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6821","nonce":482472979}]},"public_addr":"172.21.15.195:6811/482472979","cluster_addr":"172.21.15.195:6816/482472979","heartbeat_back_addr":"172.21.15.195:6823/482472979","heartbeat_front_addr":"172.21.15.195:6821/482472979","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.754003+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.043603+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.024506+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.033767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.994825+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.038300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T11:58:08.259 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.257+0000 7f90ba250700 1 -- 172.21.15.78:0/798113300 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f909406ae70 msgr2=0x7f909406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:08.259 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.257+0000 7f90ba250700 1 --2- 172.21.15.78:0/798113300 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f909406ae70 0x7f909406d330 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f90a4001eb0 tx=0x7f90a4000f80 comp rx=0 tx=0).stop 2023-10-14T11:58:08.259 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.257+0000 7f90ba250700 1 -- 172.21.15.78:0/798113300 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90b4149a60 msgr2=0x7f90b413cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:08.260 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.257+0000 7f90ba250700 1 --2- 172.21.15.78:0/798113300 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90b4149a60 0x7f90b413cbe0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f90b406fea0 tx=0x7f90ac011ed0 comp rx=0 tx=0).stop 2023-10-14T11:58:08.260 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.257+0000 7f90ba250700 1 -- 172.21.15.78:0/798113300 shutdown_connections 2023-10-14T11:58:08.260 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.257+0000 7f90ba250700 1 --2- 172.21.15.78:0/798113300 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90b4149a60 0x7f90b413cbe0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:08.260 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.257+0000 7f90ba250700 1 --2- 172.21.15.78:0/798113300 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f90a0003fe0 0x7f90b40ac350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:08.260 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.257+0000 7f90ba250700 1 --2- 172.21.15.78:0/798113300 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f909406ae70 0x7f909406d330 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:08.261 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.257+0000 7f90ba250700 1 --2- 172.21.15.78:0/798113300 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f90b40c35b0 0x7f90b413c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:08.261 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.257+0000 7f90ba250700 1 -- 172.21.15.78:0/798113300 >> 172.21.15.78:0/798113300 conn(0x7f90b4071320 msgr2=0x7f90b41dd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:08.261 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.257+0000 7f90ba250700 1 -- 172.21.15.78:0/798113300 shutdown_connections 2023-10-14T11:58:08.261 INFO:teuthology.misc.health.smithi078.stderr:2023-10-14T11:58:08.257+0000 7f90ba250700 1 -- 172.21.15.78:0/798113300 wait complete. 2023-10-14T11:58:08.270 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-10-14T11:58:08.270 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-14T11:58:08.271 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-14T11:58:08.271 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-14T11:58:08.274 INFO:tasks.ceph.mds.a:Started 2023-10-14T11:58:08.274 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-14T11:58:08.274 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-14T11:58:08.276 INFO:tasks.ceph.mds.c:Started 2023-10-14T11:58:08.277 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-14T11:58:08.277 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-14T11:58:08.280 INFO:tasks.ceph.mds.b:Started 2023-10-14T11:58:08.280 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-14T11:58:08.281 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-14T11:58:08.283 INFO:tasks.ceph.mds.d:Started 2023-10-14T11:58:08.283 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-10-14T11:58:08.284 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T11:58:08.353 INFO:tasks.ceph.mds.a.smithi078.stdout:starting mds.a at 2023-10-14T11:58:08.354 INFO:tasks.ceph.mds.c.smithi078.stdout:starting mds.c at 2023-10-14T11:58:08.369 INFO:tasks.ceph.mds.b.smithi195.stdout:starting mds.b at 2023-10-14T11:58:08.370 INFO:tasks.ceph.mds.d.smithi195.stdout:starting mds.d at 2023-10-14T11:58:08.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.410+0000 7f22e1663700 1 Processor -- start 2023-10-14T11:58:08.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.411+0000 7f22e1663700 1 -- start start 2023-10-14T11:58:08.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.411+0000 7f22e1663700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f22dc152110 0x7f22dc06fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:08.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.411+0000 7f22e1663700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f22dc0c7340 0x7f22dc0700f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:08.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.411+0000 7f22e1663700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22dc158450 con 0x7f22dc06d6b0 2023-10-14T11:58:08.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.411+0000 7f22e1663700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22dc1590d0 con 0x7f22dc0c7340 2023-10-14T11:58:08.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.411+0000 7f22e1663700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22dc158730 con 0x7f22dc152110 2023-10-14T11:58:08.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.411+0000 7f22db7fe700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f22dc0c7340 0x7f22dc0700f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:08.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.411+0000 7f22db7fe700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f22dc0c7340 0x7f22dc0700f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44202/0 (socket says 172.21.15.78:44202) 2023-10-14T11:58:08.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.411+0000 7f22db7fe700 1 -- 172.21.15.78:0/40973214 learned_addr learned my addr 172.21.15.78:0/40973214 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:08.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.412+0000 7f22daffd700 1 --2- 172.21.15.78:0/40973214 >> v2:172.21.15.195:3300/0 conn(0x7f22dc152110 0x7f22dc06fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:08.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.412+0000 7f22db7fe700 1 -- 172.21.15.78:0/40973214 >> v2:172.21.15.195:3300/0 conn(0x7f22dc152110 msgr2=0x7f22dc06fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:08.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.412+0000 7f22db7fe700 1 --2- 172.21.15.78:0/40973214 >> v2:172.21.15.195:3300/0 conn(0x7f22dc152110 0x7f22dc06fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:08.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.412+0000 7f22db7fe700 1 -- 172.21.15.78:0/40973214 >> v1:172.21.15.195:6789/0 conn(0x7f22dc06d6b0 legacy=0x7f22dc0c6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:08.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.412+0000 7f22db7fe700 1 -- 172.21.15.78:0/40973214 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22dc158a10 con 0x7f22dc0c7340 2023-10-14T11:58:08.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.412+0000 7f22db7fe700 1 --2- 172.21.15.78:0/40973214 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22dc0c7340 0x7f22dc0700f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f22d4009e50 tx=0x7f22d400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c54df04bea97c2a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:08.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.412+0000 7f22d9ffb700 1 -- 172.21.15.78:0/40973214 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d400e070 con 0x7f22dc0c7340 2023-10-14T11:58:08.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.412+0000 7f22d9ffb700 1 -- 172.21.15.78:0/40973214 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22d4003a20 con 0x7f22dc0c7340 2023-10-14T11:58:08.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.412+0000 7f22d9ffb700 1 -- 172.21.15.78:0/40973214 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d4003dd0 con 0x7f22dc0c7340 2023-10-14T11:58:08.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.412+0000 7f22e1663700 1 -- 172.21.15.78:0/40973214 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22dc0c7340 msgr2=0x7f22dc0700f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:08.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.412+0000 7f22e1663700 1 --2- 172.21.15.78:0/40973214 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22dc0c7340 0x7f22dc0700f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f22d4009e50 tx=0x7f22d400b330 comp rx=0 tx=0).stop 2023-10-14T11:58:08.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.412+0000 7f22e1663700 1 -- 172.21.15.78:0/40973214 shutdown_connections 2023-10-14T11:58:08.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.413+0000 7f22e1663700 1 --2- 172.21.15.78:0/40973214 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22dc0c7340 0x7f22dc0700f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:08.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.413+0000 7f22e1663700 1 --2- 172.21.15.78:0/40973214 >> v2:172.21.15.195:3300/0 conn(0x7f22dc152110 0x7f22dc06fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:08.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.413+0000 7f22e1663700 1 -- 172.21.15.78:0/40973214 >> 172.21.15.78:0/40973214 conn(0x7f22dc0c80d0 msgr2=0x7f22dc0c84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:08.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.413+0000 7f22e1663700 1 -- 172.21.15.78:0/40973214 shutdown_connections 2023-10-14T11:58:08.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.413+0000 7f22e1663700 1 -- 172.21.15.78:0/40973214 wait complete. 2023-10-14T11:58:08.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.414+0000 7f22e1663700 1 Processor -- start 2023-10-14T11:58:08.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.414+0000 7f22e1663700 1 -- start start 2023-10-14T11:58:08.424 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.415+0000 7f22e1663700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f22dc06d6b0 0x7f22dc0c2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:08.424 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.415+0000 7f22e1663700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22dc0c7340 0x7f22dc0a9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:08.424 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.415+0000 7f22e1663700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f22dc152110 0x7f22dc0aa530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:08.424 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.415+0000 7f22e1663700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22dc0aaa70 con 0x7f22dc0c7340 2023-10-14T11:58:08.424 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.415+0000 7f22e1663700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22dc0c2200 con 0x7f22dc152110 2023-10-14T11:58:08.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.415+0000 7f22e1663700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22dc0c59c0 con 0x7f22dc06d6b0 2023-10-14T11:58:08.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.415+0000 7f22da7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22dc0c7340 0x7f22dc0a9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:08.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.415+0000 7f22da7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22dc0c7340 0x7f22dc0a9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44206/0 (socket says 172.21.15.78:44206) 2023-10-14T11:58:08.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.415+0000 7f22da7fc700 1 -- 172.21.15.78:0/2264790807 learned_addr learned my addr 172.21.15.78:0/2264790807 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:08.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.415+0000 7f22daffd700 1 --2- 172.21.15.78:0/2264790807 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f22dc06d6b0 0x7f22dc0c2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:08.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.415+0000 7f22db7fe700 1 --2- 172.21.15.78:0/2264790807 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f22dc152110 0x7f22dc0aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:08.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.415+0000 7f22da7fc700 1 -- 172.21.15.78:0/2264790807 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f22dc06d6b0 msgr2=0x7f22dc0c2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:08.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.415+0000 7f22da7fc700 1 --2- 172.21.15.78:0/2264790807 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f22dc06d6b0 0x7f22dc0c2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:08.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.415+0000 7f22da7fc700 1 -- 172.21.15.78:0/2264790807 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f22dc152110 msgr2=0x7f22dc0aa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:08.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.415+0000 7f22da7fc700 1 --2- 172.21.15.78:0/2264790807 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f22dc152110 0x7f22dc0aa530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:08.427 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.415+0000 7f22da7fc700 1 -- 172.21.15.78:0/2264790807 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22d4009b30 con 0x7f22dc0c7340 2023-10-14T11:58:08.427 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.415+0000 7f22daffd700 1 --2- 172.21.15.78:0/2264790807 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f22dc06d6b0 0x7f22dc0c2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:08.427 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.415+0000 7f22db7fe700 1 --2- 172.21.15.78:0/2264790807 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f22dc152110 0x7f22dc0aa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:08.427 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.416+0000 7f22da7fc700 1 --2- 172.21.15.78:0/2264790807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22dc0c7340 0x7f22dc0a9ff0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f22cc00b470 tx=0x7f22cc00b830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:08.427 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.416+0000 7f22c37fe700 1 -- 172.21.15.78:0/2264790807 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22cc011830 con 0x7f22dc0c7340 2023-10-14T11:58:08.428 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.416+0000 7f22c37fe700 1 -- 172.21.15.78:0/2264790807 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22cc011990 con 0x7f22dc0c7340 2023-10-14T11:58:08.428 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.416+0000 7f22e1663700 1 -- 172.21.15.78:0/2264790807 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22dc0c4e40 con 0x7f22dc0c7340 2023-10-14T11:58:08.428 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.416+0000 7f22c37fe700 1 -- 172.21.15.78:0/2264790807 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22cc011d40 con 0x7f22dc0c7340 2023-10-14T11:58:08.428 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.416+0000 7f22e1663700 1 -- 172.21.15.78:0/2264790807 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22dc0c5400 con 0x7f22dc0c7340 2023-10-14T11:58:08.428 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.417+0000 7f22e1663700 1 -- 172.21.15.78:0/2264790807 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22dc14e9a0 con 0x7f22dc0c7340 2023-10-14T11:58:08.429 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.417+0000 7f22c37fe700 1 -- 172.21.15.78:0/2264790807 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 4) v1 ==== 89173+0+0 (secure 0 0 0) 0x7f22cc01a400 con 0x7f22dc0c7340 2023-10-14T11:58:08.429 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.417+0000 7f22c37fe700 1 --2- 172.21.15.78:0/2264790807 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f22c806ae70 0x7f22c806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:08.429 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.418+0000 7f22c37fe700 1 -- 172.21.15.78:0/2264790807 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f22cc089040 con 0x7f22dc0c7340 2023-10-14T11:58:08.429 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.418+0000 7f22daffd700 1 --2- 172.21.15.78:0/2264790807 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f22c806ae70 0x7f22c806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:08.430 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.418+0000 7f22daffd700 1 --2- 172.21.15.78:0/2264790807 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f22c806ae70 0x7f22c806d330 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f22d0006fd0 tx=0x7f22d0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:08.430 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.422+0000 7f22c37fe700 1 -- 172.21.15.78:0/2264790807 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f22dc14e9a0 con 0x7f22dc0c7340 2023-10-14T11:58:08.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.612+0000 7f22e1663700 1 -- 172.21.15.78:0/2264790807 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22dc14e9a0 con 0x7f22dc0c7340 2023-10-14T11:58:08.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.613+0000 7f22c37fe700 1 -- 172.21.15.78:0/2264790807 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+11022 (secure 0 0 0) 0x7f22dc14e9a0 con 0x7f22dc0c7340 2023-10-14T11:58:08.617 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:08.618 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":14,"fsid":"4c5dc134-3092-406a-a1e1-13e6d9e926e8","created":"2023-10-14T11:57:57.363750+0000","modified":"2023-10-14T11:58:07.984311+0000","last_up_change":"2023-10-14T11:58:06.982472+0000","last_in_change":"2023-10-14T11:57:59.821373+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"625a5fc0-9474-4eae-b80e-14c27a74ad19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6801","nonce":3884969780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6803","nonce":3884969780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6807","nonce":3884969780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6805","nonce":3884969780}]},"public_addr":"172.21.15.78:6801/3884969780","cluster_addr":"172.21.15.78:6803/3884969780","heartbeat_back_addr":"172.21.15.78:6807/3884969780","heartbeat_front_addr":"172.21.15.78:6805/3884969780","state":["exists","up"]},{"osd":1,"uuid":"49d63db1-2981-49ab-9c40-c50fc744b982","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6809","nonce":2176928406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6811","nonce":2176928406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6815","nonce":2176928406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6813","nonce":2176928406}]},"public_addr":"172.21.15.78:6809/2176928406","cluster_addr":"172.21.15.78:6811/2176928406","heartbeat_back_addr":"172.21.15.78:6815/2176928406","heartbeat_front_addr":"172.21.15.78:6813/2176928406","state":["exists","up"]},{"osd":2,"uuid":"1d7d0b29-cbd1-4717-9fe6-765a8d4998b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6825","nonce":3647044577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6827","nonce":3647044577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6831","nonce":3647044577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6829","nonce":3647044577}]},"public_addr":"172.21.15.78:6825/3647044577","cluster_addr":"172.21.15.78:6827/3647044577","heartbeat_back_addr":"172.21.15.78:6831/3647044577","heartbeat_front_addr":"172.21.15.78:6829/3647044577","state":["exists","up"]},{"osd":3,"uuid":"2f75ded1-e9e2-40e7-99ae-45e1e425e1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6817","nonce":3584112288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6819","nonce":3584112288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6823","nonce":3584112288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6821","nonce":3584112288}]},"public_addr":"172.21.15.78:6817/3584112288","cluster_addr":"172.21.15.78:6819/3584112288","heartbeat_back_addr":"172.21.15.78:6823/3584112288","heartbeat_front_addr":"172.21.15.78:6821/3584112288","state":["exists","up"]},{"osd":4,"uuid":"5130b432-8358-46cb-8a31-3064c976f6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6801","nonce":3732756301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6803","nonce":3732756301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6807","nonce":3732756301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6805","nonce":3732756301}]},"public_addr":"172.21.15.195:6801/3732756301","cluster_addr":"172.21.15.195:6803/3732756301","heartbeat_back_addr":"172.21.15.195:6807/3732756301","heartbeat_front_addr":"172.21.15.195:6805/3732756301","state":["exists","up"]},{"osd":5,"uuid":"60e234a9-a506-4e40-9c19-49a58f3cf28c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6825","nonce":2857706411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6827","nonce":2857706411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6831","nonce":2857706411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6829","nonce":2857706411}]},"public_addr":"172.21.15.195:6825/2857706411","cluster_addr":"172.21.15.195:6827/2857706411","heartbeat_back_addr":"172.21.15.195:6831/2857706411","heartbeat_front_addr":"172.21.15.195:6829/2857706411","state":["exists","up"]},{"osd":6,"uuid":"df7d2dd6-26b6-4eb1-96e5-94d1f3955b33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6810","nonce":72554374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6813","nonce":72554374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6820","nonce":72554374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6817","nonce":72554374}]},"public_addr":"172.21.15.195:6810/72554374","cluster_addr":"172.21.15.195:6813/72554374","heartbeat_back_addr":"172.21.15.195:6820/72554374","heartbeat_front_addr":"172.21.15.195:6817/72554374","state":["exists","up"]},{"osd":7,"uuid":"47a9d4de-f56d-4aa4-9900-c4a9cf9b2f97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6809","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6811","nonce":482472979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6816","nonce":482472979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6823","nonce":482472979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6821","nonce":482472979}]},"public_addr":"172.21.15.195:6811/482472979","cluster_addr":"172.21.15.195:6816/482472979","heartbeat_back_addr":"172.21.15.195:6823/482472979","heartbeat_front_addr":"172.21.15.195:6821/482472979","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.754003+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.043603+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.024506+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.033767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.994825+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.038300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T11:58:08.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.615+0000 7f22e1663700 1 -- 172.21.15.78:0/2264790807 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f22c806ae70 msgr2=0x7f22c806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:08.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.615+0000 7f22e1663700 1 --2- 172.21.15.78:0/2264790807 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f22c806ae70 0x7f22c806d330 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f22d0006fd0 tx=0x7f22d0008040 comp rx=0 tx=0).stop 2023-10-14T11:58:08.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.615+0000 7f22e1663700 1 -- 172.21.15.78:0/2264790807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22dc0c7340 msgr2=0x7f22dc0a9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:08.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.615+0000 7f22e1663700 1 --2- 172.21.15.78:0/2264790807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22dc0c7340 0x7f22dc0a9ff0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f22cc00b470 tx=0x7f22cc00b830 comp rx=0 tx=0).stop 2023-10-14T11:58:08.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.615+0000 7f22e1663700 1 -- 172.21.15.78:0/2264790807 shutdown_connections 2023-10-14T11:58:08.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.615+0000 7f22e1663700 1 --2- 172.21.15.78:0/2264790807 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f22c806ae70 0x7f22c806d330 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:08.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.615+0000 7f22e1663700 1 --2- 172.21.15.78:0/2264790807 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f22dc152110 0x7f22dc0aa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:08.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.615+0000 7f22e1663700 1 --2- 172.21.15.78:0/2264790807 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f22dc06d6b0 0x7f22dc0c2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:08.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.615+0000 7f22e1663700 1 --2- 172.21.15.78:0/2264790807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22dc0c7340 0x7f22dc0a9ff0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:08.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.615+0000 7f22e1663700 1 -- 172.21.15.78:0/2264790807 >> 172.21.15.78:0/2264790807 conn(0x7f22dc0c80d0 msgr2=0x7f22dc156bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:08.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.615+0000 7f22e1663700 1 -- 172.21.15.78:0/2264790807 shutdown_connections 2023-10-14T11:58:08.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.616+0000 7f22e1663700 1 -- 172.21.15.78:0/2264790807 wait complete. 2023-10-14T11:58:08.629 INFO:tasks.cephfs.filesystem.ceph_manager:[] 2023-10-14T11:58:08.630 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T11:58:08.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.748+0000 7fb7b099e700 1 Processor -- start 2023-10-14T11:58:08.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.748+0000 7fb7b099e700 1 -- start start 2023-10-14T11:58:08.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.749+0000 7fb7b099e700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb7ac06ab40 0x7fb7ac06f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:08.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.749+0000 7fb7b099e700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb7ac0c0350 0x7fb7ac13d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:08.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.749+0000 7fb7b099e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7ac06b630 con 0x7fb7ac1554e0 2023-10-14T11:58:08.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.749+0000 7fb7b099e700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7ac0c49f0 con 0x7fb7ac06ab40 2023-10-14T11:58:08.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.749+0000 7fb7b099e700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ac0726a0 con 0x7fb7ac0c0350 2023-10-14T11:58:08.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.749+0000 7fb7aa59c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb7ac06ab40 0x7fb7ac06f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:08.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.749+0000 7fb7aad9d700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb7ac0c0350 0x7fb7ac13d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:08.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.749+0000 7fb7aa59c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb7ac06ab40 0x7fb7ac06f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.78:46968/0 (socket says 172.21.15.78:46968) 2023-10-14T11:58:08.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.749+0000 7fb7aad9d700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb7ac0c0350 0x7fb7ac13d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.78:43904/0 (socket says 172.21.15.78:43904) 2023-10-14T11:58:08.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.749+0000 7fb7a9d9b700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fb7ac1554e0 0x7fb7ac06af20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.78:38908/0 (socket says 172.21.15.78:38908) 2023-10-14T11:58:08.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.749+0000 7fb7aa59c700 1 -- 172.21.15.78:0/1925818376 learned_addr learned my addr 172.21.15.78:0/1925818376 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:08.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.749+0000 7fb7aa59c700 1 -- 172.21.15.78:0/1925818376 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb7ac0c0350 msgr2=0x7fb7ac13d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:08.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.749+0000 7fb7aa59c700 1 --2- 172.21.15.78:0/1925818376 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb7ac0c0350 0x7fb7ac13d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:08.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.749+0000 7fb7aa59c700 1 -- 172.21.15.78:0/1925818376 >> v1:172.21.15.195:6789/0 conn(0x7fb7ac1554e0 legacy=0x7fb7ac06af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:08.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.749+0000 7fb7a959a700 1 -- 172.21.15.78:0/1925818376 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3887474038 0 0) 0x7fb7ac06b630 con 0x7fb7ac1554e0 2023-10-14T11:58:08.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.749+0000 7fb7aa59c700 1 -- 172.21.15.78:0/1925818376 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7ac072980 con 0x7fb7ac06ab40 2023-10-14T11:58:08.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.749+0000 7fb7aad9d700 1 --2- 172.21.15.78:0/1925818376 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb7ac0c0350 0x7fb7ac13d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T11:58:08.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.750+0000 7fb7aa59c700 1 --2- 172.21.15.78:0/1925818376 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb7ac06ab40 0x7fb7ac06f1e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb7a00098f0 tx=0x7fb7a0009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d665e5b31e026b26 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:08.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.750+0000 7fb7a959a700 1 -- 172.21.15.78:0/1925818376 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7a0011830 con 0x7fb7ac06ab40 2023-10-14T11:58:08.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.750+0000 7fb7a959a700 1 -- 172.21.15.78:0/1925818376 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7a0011990 con 0x7fb7ac06ab40 2023-10-14T11:58:08.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.750+0000 7fb7a959a700 1 -- 172.21.15.78:0/1925818376 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7a0011d20 con 0x7fb7ac06ab40 2023-10-14T11:58:08.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.750+0000 7fb7b099e700 1 -- 172.21.15.78:0/1925818376 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb7ac06ab40 msgr2=0x7fb7ac06f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:08.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.750+0000 7fb7b099e700 1 --2- 172.21.15.78:0/1925818376 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb7ac06ab40 0x7fb7ac06f1e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb7a00098f0 tx=0x7fb7a0009c00 comp rx=0 tx=0).stop 2023-10-14T11:58:08.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.750+0000 7fb7b099e700 1 -- 172.21.15.78:0/1925818376 shutdown_connections 2023-10-14T11:58:08.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.750+0000 7fb7b099e700 1 --2- 172.21.15.78:0/1925818376 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb7ac0c0350 0x7fb7ac13d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:08.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.750+0000 7fb7b099e700 1 --2- 172.21.15.78:0/1925818376 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb7ac06ab40 0x7fb7ac06f1e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:08.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.750+0000 7fb7b099e700 1 -- 172.21.15.78:0/1925818376 >> 172.21.15.78:0/1925818376 conn(0x7fb7ac0c3ac0 msgr2=0x7fb7ac0c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:08.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.750+0000 7fb7b099e700 1 -- 172.21.15.78:0/1925818376 shutdown_connections 2023-10-14T11:58:08.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.750+0000 7fb7b099e700 1 -- 172.21.15.78:0/1925818376 wait complete. 2023-10-14T11:58:08.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.752+0000 7fb7b099e700 1 Processor -- start 2023-10-14T11:58:08.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.752+0000 7fb7b099e700 1 -- start start 2023-10-14T11:58:08.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.752+0000 7fb7b099e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7ac06ab40 0x7fb7ac134800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:08.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.752+0000 7fb7b099e700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb7ac0c0350 0x7fb7ac1573b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:08.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.752+0000 7fb7b099e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb7ac1554e0 0x7fb7ac1578f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:08.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.752+0000 7fb7b099e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ac151a40 con 0x7fb7ac06ab40 2023-10-14T11:58:08.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.752+0000 7fb7b099e700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ac151150 con 0x7fb7ac1554e0 2023-10-14T11:58:08.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.752+0000 7fb7b099e700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ac134d40 con 0x7fb7ac0c0350 2023-10-14T11:58:08.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.752+0000 7fb7aa59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7ac06ab40 0x7fb7ac134800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:08.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.752+0000 7fb7aa59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7ac06ab40 0x7fb7ac134800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44236/0 (socket says 172.21.15.78:44236) 2023-10-14T11:58:08.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.752+0000 7fb7aa59c700 1 -- 172.21.15.78:0/2094623345 learned_addr learned my addr 172.21.15.78:0/2094623345 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:08.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.753+0000 7fb7aad9d700 1 --2- 172.21.15.78:0/2094623345 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb7ac1554e0 0x7fb7ac1578f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:08.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.753+0000 7fb7a9d9b700 1 --2- 172.21.15.78:0/2094623345 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb7ac0c0350 0x7fb7ac1573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:08.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.753+0000 7fb7aa59c700 1 -- 172.21.15.78:0/2094623345 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb7ac0c0350 msgr2=0x7fb7ac1573b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:08.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.753+0000 7fb7aa59c700 1 --2- 172.21.15.78:0/2094623345 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb7ac0c0350 0x7fb7ac1573b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:08.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.753+0000 7fb7aa59c700 1 -- 172.21.15.78:0/2094623345 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb7ac1554e0 msgr2=0x7fb7ac1578f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:08.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.753+0000 7fb7aa59c700 1 --2- 172.21.15.78:0/2094623345 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb7ac1554e0 0x7fb7ac1578f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:08.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.753+0000 7fb7aa59c700 1 -- 172.21.15.78:0/2094623345 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a4009a70 con 0x7fb7ac06ab40 2023-10-14T11:58:08.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.753+0000 7fb7aa59c700 1 --2- 172.21.15.78:0/2094623345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7ac06ab40 0x7fb7ac134800 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb7a0009c40 tx=0x7fb7a0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:08.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.753+0000 7fb79affd700 1 -- 172.21.15.78:0/2094623345 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7a00043c0 con 0x7fb7ac06ab40 2023-10-14T11:58:08.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.753+0000 7fb79affd700 1 -- 172.21.15.78:0/2094623345 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7a001f070 con 0x7fb7ac06ab40 2023-10-14T11:58:08.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.753+0000 7fb7b099e700 1 -- 172.21.15.78:0/2094623345 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7a0009550 con 0x7fb7ac06ab40 2023-10-14T11:58:08.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.753+0000 7fb79affd700 1 -- 172.21.15.78:0/2094623345 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7a00036a0 con 0x7fb7ac06ab40 2023-10-14T11:58:08.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.753+0000 7fb7b099e700 1 -- 172.21.15.78:0/2094623345 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7ac1513f0 con 0x7fb7ac06ab40 2023-10-14T11:58:08.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.754+0000 7fb7b099e700 1 -- 172.21.15.78:0/2094623345 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7ac074b80 con 0x7fb7ac06ab40 2023-10-14T11:58:08.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.754+0000 7fb79affd700 1 -- 172.21.15.78:0/2094623345 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 4) v1 ==== 89173+0+0 (secure 0 0 0) 0x7fb7a0021030 con 0x7fb7ac06ab40 2023-10-14T11:58:08.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.755+0000 7fb79affd700 1 --2- 172.21.15.78:0/2094623345 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb78c06ae70 0x7fb78c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:08.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.755+0000 7fb79affd700 1 -- 172.21.15.78:0/2094623345 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4471+0+0 (secure 0 0 0) 0x7fb7a008a2b0 con 0x7fb7ac06ab40 2023-10-14T11:58:08.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.755+0000 7fb7a9d9b700 1 --2- 172.21.15.78:0/2094623345 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb78c06ae70 0x7fb78c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:08.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.755+0000 7fb7a9d9b700 1 --2- 172.21.15.78:0/2094623345 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb78c06ae70 0x7fb78c06d330 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac1546b0 tx=0x7fb79c007e70 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:08.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.759+0000 7fb79affd700 1 -- 172.21.15.78:0/2094623345 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb7a00559a0 con 0x7fb7ac06ab40 2023-10-14T11:58:08.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:08.992+0000 7fb7b099e700 1 -- 172.21.15.78:0/2094623345 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7ac066ea0 con 0x7fb7ac06ab40 2023-10-14T11:58:09.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:09.000+0000 7fb79affd700 1 -- 172.21.15.78:0/2094623345 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1321 (secure 0 0 0) 0x7fb7a0015090 con 0x7fb7ac06ab40 2023-10-14T11:58:09.003 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:09.003 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/920300787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":920300787},{"type":"v1","addr":"172.21.15.78:6837","nonce":920300787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2023-10-14T11:58:09.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:09.003+0000 7fb7b099e700 1 -- 172.21.15.78:0/2094623345 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb78c06ae70 msgr2=0x7fb78c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:09.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:09.003+0000 7fb7b099e700 1 --2- 172.21.15.78:0/2094623345 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb78c06ae70 0x7fb78c06d330 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac1546b0 tx=0x7fb79c007e70 comp rx=0 tx=0).stop 2023-10-14T11:58:09.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:09.003+0000 7fb7b099e700 1 -- 172.21.15.78:0/2094623345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7ac06ab40 msgr2=0x7fb7ac134800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:09.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:09.003+0000 7fb7b099e700 1 --2- 172.21.15.78:0/2094623345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7ac06ab40 0x7fb7ac134800 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb7a0009c40 tx=0x7fb7a0004750 comp rx=0 tx=0).stop 2023-10-14T11:58:09.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:09.003+0000 7fb7b099e700 1 -- 172.21.15.78:0/2094623345 shutdown_connections 2023-10-14T11:58:09.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:09.003+0000 7fb7b099e700 1 --2- 172.21.15.78:0/2094623345 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb78c06ae70 0x7fb78c06d330 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:09.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:09.003+0000 7fb7b099e700 1 --2- 172.21.15.78:0/2094623345 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb7ac1554e0 0x7fb7ac1578f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:09.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:09.003+0000 7fb7b099e700 1 --2- 172.21.15.78:0/2094623345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7ac06ab40 0x7fb7ac134800 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:09.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:09.003+0000 7fb7b099e700 1 --2- 172.21.15.78:0/2094623345 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb7ac0c0350 0x7fb7ac1573b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:09.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:09.003+0000 7fb7b099e700 1 -- 172.21.15.78:0/2094623345 >> 172.21.15.78:0/2094623345 conn(0x7fb7ac0c3ac0 msgr2=0x7fb7ac13ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:09.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:09.003+0000 7fb7b099e700 1 -- 172.21.15.78:0/2094623345 shutdown_connections 2023-10-14T11:58:09.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:09.003+0000 7fb7b099e700 1 -- 172.21.15.78:0/2094623345 wait complete. 2023-10-14T11:58:09.010 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 2 2023-10-14T11:58:11.017 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T11:58:11.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.136+0000 7f6eaf825700 1 Processor -- start 2023-10-14T11:58:11.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.137+0000 7f6eaf825700 1 -- start start 2023-10-14T11:58:11.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.137+0000 7f6eaf825700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6ea806ab40 0x7f6ea806f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:11.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.137+0000 7f6eaf825700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ea81554e0 0x7f6ea806af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:11.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.137+0000 7f6eaf825700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea80c49f0 con 0x7f6ea80c0350 2023-10-14T11:58:11.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.137+0000 7f6eaf825700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ea80726a0 con 0x7f6ea806ab40 2023-10-14T11:58:11.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.137+0000 7f6eaf825700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea8072980 con 0x7f6ea81554e0 2023-10-14T11:58:11.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.138+0000 7f6ead5c1700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6ea806ab40 0x7f6ea806f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:11.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.138+0000 7f6eacdc0700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ea81554e0 0x7f6ea806af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:11.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.138+0000 7f6eacdc0700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ea81554e0 0x7f6ea806af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.78:43946/0 (socket says 172.21.15.78:43946) 2023-10-14T11:58:11.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.138+0000 7f6eaddc2700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f6ea80c0350 0x7f6ea813d7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.78:38956/0 (socket says 172.21.15.78:38956) 2023-10-14T11:58:11.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.138+0000 7f6eacdc0700 1 -- 172.21.15.78:0/845884911 learned_addr learned my addr 172.21.15.78:0/845884911 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:11.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.138+0000 7f6ead5c1700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6ea806ab40 0x7f6ea806f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.78:47012/0 (socket says 172.21.15.78:47012) 2023-10-14T11:58:11.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.138+0000 7f6eacdc0700 1 -- 172.21.15.78:0/845884911 >> v2:172.21.15.195:3300/0 conn(0x7f6ea806ab40 msgr2=0x7f6ea806f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:11.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.138+0000 7f6eacdc0700 1 --2- 172.21.15.78:0/845884911 >> v2:172.21.15.195:3300/0 conn(0x7f6ea806ab40 0x7f6ea806f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:11.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.138+0000 7f6eacdc0700 1 -- 172.21.15.78:0/845884911 >> v1:172.21.15.195:6789/0 conn(0x7f6ea80c0350 legacy=0x7f6ea813d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:11.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.138+0000 7f6eacdc0700 1 -- 172.21.15.78:0/845884911 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea8154a00 con 0x7f6ea81554e0 2023-10-14T11:58:11.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.138+0000 7f6ead5c1700 1 --2- 172.21.15.78:0/845884911 >> v2:172.21.15.195:3300/0 conn(0x7f6ea806ab40 0x7f6ea806f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T11:58:11.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.138+0000 7f6eacdc0700 1 --2- 172.21.15.78:0/845884911 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ea81554e0 0x7f6ea806af20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6ea40098f0 tx=0x7f6ea4009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c3b6fe7f3746c73 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:11.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.138+0000 7f6e9ffff700 1 -- 172.21.15.78:0/845884911 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea4011830 con 0x7f6ea81554e0 2023-10-14T11:58:11.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.138+0000 7f6e9ffff700 1 -- 172.21.15.78:0/845884911 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ea4011990 con 0x7f6ea81554e0 2023-10-14T11:58:11.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.138+0000 7f6e9ffff700 1 -- 172.21.15.78:0/845884911 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea4011d20 con 0x7f6ea81554e0 2023-10-14T11:58:11.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.139+0000 7f6eaf825700 1 -- 172.21.15.78:0/845884911 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ea81554e0 msgr2=0x7f6ea806af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:11.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.139+0000 7f6eaf825700 1 --2- 172.21.15.78:0/845884911 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ea81554e0 0x7f6ea806af20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6ea40098f0 tx=0x7f6ea4009c00 comp rx=0 tx=0).stop 2023-10-14T11:58:11.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.139+0000 7f6eaf825700 1 -- 172.21.15.78:0/845884911 shutdown_connections 2023-10-14T11:58:11.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.139+0000 7f6eaf825700 1 --2- 172.21.15.78:0/845884911 >> v2:172.21.15.195:3300/0 conn(0x7f6ea806ab40 0x7f6ea806f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:11.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.139+0000 7f6eaf825700 1 --2- 172.21.15.78:0/845884911 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ea81554e0 0x7f6ea806af20 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:11.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.139+0000 7f6eaf825700 1 -- 172.21.15.78:0/845884911 >> 172.21.15.78:0/845884911 conn(0x7f6ea80c3ac0 msgr2=0x7f6ea80c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:11.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.139+0000 7f6eaf825700 1 -- 172.21.15.78:0/845884911 shutdown_connections 2023-10-14T11:58:11.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.139+0000 7f6eaf825700 1 -- 172.21.15.78:0/845884911 wait complete. 2023-10-14T11:58:11.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.140+0000 7f6eaf825700 1 Processor -- start 2023-10-14T11:58:11.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.140+0000 7f6eaf825700 1 -- start start 2023-10-14T11:58:11.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.141+0000 7f6eaf825700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ea806ab40 0x7f6ea81edce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:11.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.141+0000 7f6eaf825700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ea80c0350 0x7f6ea80c3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:11.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.141+0000 7f6eaf825700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ea81554e0 0x7f6ea80ab7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:11.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.141+0000 7f6eaf825700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea81563b0 con 0x7f6ea80c0350 2023-10-14T11:58:11.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.141+0000 7f6eaf825700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea81571d0 con 0x7f6ea806ab40 2023-10-14T11:58:11.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.141+0000 7f6eaf825700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea81ee220 con 0x7f6ea81554e0 2023-10-14T11:58:11.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.141+0000 7f6eacdc0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ea80c0350 0x7f6ea80c3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:11.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.141+0000 7f6ead5c1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ea806ab40 0x7f6ea81edce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:11.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.141+0000 7f6eacdc0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ea80c0350 0x7f6ea80c3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44282/0 (socket says 172.21.15.78:44282) 2023-10-14T11:58:11.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.141+0000 7f6eacdc0700 1 -- 172.21.15.78:0/2442972028 learned_addr learned my addr 172.21.15.78:0/2442972028 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:11.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.141+0000 7f6eaddc2700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ea81554e0 0x7f6ea80ab7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:11.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.141+0000 7f6eacdc0700 1 -- 172.21.15.78:0/2442972028 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ea81554e0 msgr2=0x7f6ea80ab7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:11.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.141+0000 7f6eacdc0700 1 --2- 172.21.15.78:0/2442972028 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ea81554e0 0x7f6ea80ab7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:11.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.141+0000 7f6eacdc0700 1 -- 172.21.15.78:0/2442972028 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ea806ab40 msgr2=0x7f6ea81edce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:11.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.141+0000 7f6eacdc0700 1 --2- 172.21.15.78:0/2442972028 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ea806ab40 0x7f6ea81edce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:11.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.141+0000 7f6eacdc0700 1 -- 172.21.15.78:0/2442972028 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e980094a0 con 0x7f6ea80c0350 2023-10-14T11:58:11.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.141+0000 7f6eacdc0700 1 --2- 172.21.15.78:0/2442972028 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ea80c0350 0x7f6ea80c3250 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4009c40 tx=0x7f6ea4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:11.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.141+0000 7f6e9dffb700 1 -- 172.21.15.78:0/2442972028 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea4004510 con 0x7f6ea80c0350 2023-10-14T11:58:11.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.141+0000 7f6e9dffb700 1 -- 172.21.15.78:0/2442972028 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ea401e070 con 0x7f6ea80c0350 2023-10-14T11:58:11.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.141+0000 7f6eaf825700 1 -- 172.21.15.78:0/2442972028 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ea4009550 con 0x7f6ea80c0350 2023-10-14T11:58:11.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.141+0000 7f6e9dffb700 1 -- 172.21.15.78:0/2442972028 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea40038e0 con 0x7f6ea80c0350 2023-10-14T11:58:11.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.142+0000 7f6eaf825700 1 -- 172.21.15.78:0/2442972028 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ea81560d0 con 0x7f6ea80c0350 2023-10-14T11:58:11.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.143+0000 7f6eaf825700 1 -- 172.21.15.78:0/2442972028 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ea8157340 con 0x7f6ea80c0350 2023-10-14T11:58:11.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.143+0000 7f6e9dffb700 1 -- 172.21.15.78:0/2442972028 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 4) v1 ==== 89173+0+0 (secure 0 0 0) 0x7f6ea4003a40 con 0x7f6ea80c0350 2023-10-14T11:58:11.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.143+0000 7f6e9dffb700 1 --2- 172.21.15.78:0/2442972028 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f6e8c06aea0 0x7f6e8c06d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:11.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.143+0000 7f6e9dffb700 1 -- 172.21.15.78:0/2442972028 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6ea4089f70 con 0x7f6ea80c0350 2023-10-14T11:58:11.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.144+0000 7f6ead5c1700 1 --2- 172.21.15.78:0/2442972028 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f6e8c06aea0 0x7f6e8c06d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:11.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.144+0000 7f6ead5c1700 1 --2- 172.21.15.78:0/2442972028 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f6e8c06aea0 0x7f6e8c06d360 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6e98009d70 tx=0x7f6e98009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:11.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.148+0000 7f6e9dffb700 1 -- 172.21.15.78:0/2442972028 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6ea4055460 con 0x7f6ea80c0350 2023-10-14T11:58:11.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.387+0000 7f6eaf825700 1 -- 172.21.15.78:0/2442972028 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ea8074b80 con 0x7f6ea80c0350 2023-10-14T11:58:11.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.387+0000 7f6e9dffb700 1 -- 172.21.15.78:0/2442972028 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3691 (secure 0 0 0) 0x7f6ea4020090 con 0x7f6ea80c0350 2023-10-14T11:58:11.390 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:11.390 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/353043244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":353043244},{"type":"v1","addr":"172.21.15.78:6836","nonce":353043244}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6834/3389817343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":3389817343},{"type":"v1","addr":"172.21.15.195:6834","nonce":3389817343}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4267,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/460176701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6833","nonce":460176701},{"type":"v1","addr":"172.21.15.195:6835","nonce":460176701}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/920300787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":920300787},{"type":"v1","addr":"172.21.15.78:6837","nonce":920300787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2023-10-14T11:58:11.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.389+0000 7f6eaf825700 1 -- 172.21.15.78:0/2442972028 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f6e8c06aea0 msgr2=0x7f6e8c06d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:11.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.389+0000 7f6eaf825700 1 --2- 172.21.15.78:0/2442972028 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f6e8c06aea0 0x7f6e8c06d360 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6e98009d70 tx=0x7f6e98009210 comp rx=0 tx=0).stop 2023-10-14T11:58:11.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.389+0000 7f6eaf825700 1 -- 172.21.15.78:0/2442972028 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ea80c0350 msgr2=0x7f6ea80c3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:11.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.389+0000 7f6eaf825700 1 --2- 172.21.15.78:0/2442972028 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ea80c0350 0x7f6ea80c3250 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4009c40 tx=0x7f6ea4004750 comp rx=0 tx=0).stop 2023-10-14T11:58:11.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.389+0000 7f6eaf825700 1 -- 172.21.15.78:0/2442972028 shutdown_connections 2023-10-14T11:58:11.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.389+0000 7f6eaf825700 1 --2- 172.21.15.78:0/2442972028 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ea81554e0 0x7f6ea80ab7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:11.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.389+0000 7f6eaf825700 1 --2- 172.21.15.78:0/2442972028 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f6e8c06aea0 0x7f6e8c06d360 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:11.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.389+0000 7f6eaf825700 1 --2- 172.21.15.78:0/2442972028 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ea806ab40 0x7f6ea81edce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:11.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.389+0000 7f6eaf825700 1 --2- 172.21.15.78:0/2442972028 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ea80c0350 0x7f6ea80c3250 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:11.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.389+0000 7f6eaf825700 1 -- 172.21.15.78:0/2442972028 >> 172.21.15.78:0/2442972028 conn(0x7f6ea80c3ac0 msgr2=0x7f6ea813c9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:11.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.390+0000 7f6eaf825700 1 -- 172.21.15.78:0/2442972028 shutdown_connections 2023-10-14T11:58:11.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.390+0000 7f6eaf825700 1 -- 172.21.15.78:0/2442972028 wait complete. 2023-10-14T11:58:11.394 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 3 2023-10-14T11:58:11.404 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T11:58:11.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.528+0000 7f82f65e1700 1 Processor -- start 2023-10-14T11:58:11.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.528+0000 7f82f65e1700 1 -- start start 2023-10-14T11:58:11.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.529+0000 7f82f65e1700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f82f006ab40 0x7f82f006f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:11.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.529+0000 7f82f65e1700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f82f01554e0 0x7f82f006af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:11.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.529+0000 7f82f65e1700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f82f00c0350 0x7f82f013d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:11.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.529+0000 7f82f65e1700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82f00c49f0 con 0x7f82f00c0350 2023-10-14T11:58:11.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.529+0000 7f82f65e1700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82f00726a0 con 0x7f82f01554e0 2023-10-14T11:58:11.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.529+0000 7f82f65e1700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f0072980 con 0x7f82f006ab40 2023-10-14T11:58:11.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.529+0000 7f82f4b7e700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f82f00c0350 0x7f82f013d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:11.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.529+0000 7f82f4b7e700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f82f00c0350 0x7f82f013d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44302/0 (socket says 172.21.15.78:44302) 2023-10-14T11:58:11.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.529+0000 7f82f4b7e700 1 -- 172.21.15.78:0/4000883109 learned_addr learned my addr 172.21.15.78:0/4000883109 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:11.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.529+0000 7f82effff700 1 --2- 172.21.15.78:0/4000883109 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f82f006ab40 0x7f82f006f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:11.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.529+0000 7f82ef7fe700 1 --2- 172.21.15.78:0/4000883109 >> v2:172.21.15.195:3300/0 conn(0x7f82f01554e0 0x7f82f006af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:11.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.529+0000 7f82f4b7e700 1 -- 172.21.15.78:0/4000883109 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f82f006ab40 msgr2=0x7f82f006f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:11.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.529+0000 7f82f4b7e700 1 --2- 172.21.15.78:0/4000883109 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f82f006ab40 0x7f82f006f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:11.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.529+0000 7f82f4b7e700 1 -- 172.21.15.78:0/4000883109 >> v2:172.21.15.195:3300/0 conn(0x7f82f01554e0 msgr2=0x7f82f006af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:11.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.529+0000 7f82f4b7e700 1 --2- 172.21.15.78:0/4000883109 >> v2:172.21.15.195:3300/0 conn(0x7f82f01554e0 0x7f82f006af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:11.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.529+0000 7f82f4b7e700 1 -- 172.21.15.78:0/4000883109 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f0154a00 con 0x7f82f00c0350 2023-10-14T11:58:11.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.529+0000 7f82ef7fe700 1 --2- 172.21.15.78:0/4000883109 >> v2:172.21.15.195:3300/0 conn(0x7f82f01554e0 0x7f82f006af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:11.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.530+0000 7f82f4b7e700 1 --2- 172.21.15.78:0/4000883109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82f00c0350 0x7f82f013d7b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f82f0073020 tx=0x7f82e800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef76c181fd5d3d58 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:11.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.530+0000 7f82eeffd700 1 -- 172.21.15.78:0/4000883109 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e800e070 con 0x7f82f00c0350 2023-10-14T11:58:11.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.530+0000 7f82eeffd700 1 -- 172.21.15.78:0/4000883109 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e800b9c0 con 0x7f82f00c0350 2023-10-14T11:58:11.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.530+0000 7f82eeffd700 1 -- 172.21.15.78:0/4000883109 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e800bd70 con 0x7f82f00c0350 2023-10-14T11:58:11.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.530+0000 7f82f65e1700 1 -- 172.21.15.78:0/4000883109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82f00c0350 msgr2=0x7f82f013d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:11.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.530+0000 7f82f65e1700 1 --2- 172.21.15.78:0/4000883109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82f00c0350 0x7f82f013d7b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f82f0073020 tx=0x7f82e800b330 comp rx=0 tx=0).stop 2023-10-14T11:58:11.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.530+0000 7f82f65e1700 1 -- 172.21.15.78:0/4000883109 shutdown_connections 2023-10-14T11:58:11.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.530+0000 7f82f65e1700 1 --2- 172.21.15.78:0/4000883109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82f00c0350 0x7f82f013d7b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:11.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.530+0000 7f82f65e1700 1 --2- 172.21.15.78:0/4000883109 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f82f006ab40 0x7f82f006f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:11.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.530+0000 7f82f65e1700 1 --2- 172.21.15.78:0/4000883109 >> v2:172.21.15.195:3300/0 conn(0x7f82f01554e0 0x7f82f006af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:11.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.530+0000 7f82f65e1700 1 -- 172.21.15.78:0/4000883109 >> 172.21.15.78:0/4000883109 conn(0x7f82f00c3ac0 msgr2=0x7f82f00c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:11.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.530+0000 7f82f65e1700 1 -- 172.21.15.78:0/4000883109 shutdown_connections 2023-10-14T11:58:11.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.530+0000 7f82f65e1700 1 -- 172.21.15.78:0/4000883109 wait complete. 2023-10-14T11:58:11.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.532+0000 7f82f65e1700 1 Processor -- start 2023-10-14T11:58:11.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.532+0000 7f82f65e1700 1 -- start start 2023-10-14T11:58:11.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.533+0000 7f82f65e1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f82f006ab40 0x7f82f0138580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:11.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.533+0000 7f82f65e1700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f82f00c0350 0x7f82f00c4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:11.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.533+0000 7f82f65e1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82f01554e0 0x7f82f0134360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:11.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.533+0000 7f82f65e1700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f01374a0 con 0x7f82f01554e0 2023-10-14T11:58:11.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.533+0000 7f82f65e1700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f0158e60 con 0x7f82f006ab40 2023-10-14T11:58:11.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.533+0000 7f82f65e1700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f01579b0 con 0x7f82f00c0350 2023-10-14T11:58:11.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.533+0000 7f82f4b7e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82f01554e0 0x7f82f0134360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:11.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.533+0000 7f82f4b7e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82f01554e0 0x7f82f0134360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44306/0 (socket says 172.21.15.78:44306) 2023-10-14T11:58:11.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.533+0000 7f82f4b7e700 1 -- 172.21.15.78:0/2575760923 learned_addr learned my addr 172.21.15.78:0/2575760923 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:11.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.533+0000 7f82effff700 1 --2- 172.21.15.78:0/2575760923 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f82f006ab40 0x7f82f0138580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:11.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.533+0000 7f82ef7fe700 1 --2- 172.21.15.78:0/2575760923 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f82f00c0350 0x7f82f00c4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:11.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.533+0000 7f82f4b7e700 1 -- 172.21.15.78:0/2575760923 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f82f00c0350 msgr2=0x7f82f00c4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:11.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.533+0000 7f82f4b7e700 1 --2- 172.21.15.78:0/2575760923 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f82f00c0350 0x7f82f00c4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:11.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.533+0000 7f82f4b7e700 1 -- 172.21.15.78:0/2575760923 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f82f006ab40 msgr2=0x7f82f0138580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:11.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.533+0000 7f82f4b7e700 1 --2- 172.21.15.78:0/2575760923 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f82f006ab40 0x7f82f0138580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:11.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.533+0000 7f82f4b7e700 1 -- 172.21.15.78:0/2575760923 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e8009b30 con 0x7f82f01554e0 2023-10-14T11:58:11.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.533+0000 7f82f4b7e700 1 --2- 172.21.15.78:0/2575760923 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82f01554e0 0x7f82f0134360 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f82f00ad290 tx=0x7f82e8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:11.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.534+0000 7f82ecff9700 1 -- 172.21.15.78:0/2575760923 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e80099b0 con 0x7f82f01554e0 2023-10-14T11:58:11.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.534+0000 7f82ecff9700 1 -- 172.21.15.78:0/2575760923 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e8004250 con 0x7f82f01554e0 2023-10-14T11:58:11.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.534+0000 7f82f65e1700 1 -- 172.21.15.78:0/2575760923 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82f01ede60 con 0x7f82f01554e0 2023-10-14T11:58:11.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.534+0000 7f82ecff9700 1 -- 172.21.15.78:0/2575760923 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e8004600 con 0x7f82f01554e0 2023-10-14T11:58:11.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.534+0000 7f82f65e1700 1 -- 172.21.15.78:0/2575760923 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f01ee140 con 0x7f82f01554e0 2023-10-14T11:58:11.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.535+0000 7f82f65e1700 1 -- 172.21.15.78:0/2575760923 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82f0157e90 con 0x7f82f01554e0 2023-10-14T11:58:11.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.535+0000 7f82ecff9700 1 -- 172.21.15.78:0/2575760923 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 4) v1 ==== 89173+0+0 (secure 0 0 0) 0x7f82e8004760 con 0x7f82f01554e0 2023-10-14T11:58:11.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.536+0000 7f82ecff9700 1 --2- 172.21.15.78:0/2575760923 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f82dc06ae70 0x7f82dc06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:11.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.536+0000 7f82effff700 1 --2- 172.21.15.78:0/2575760923 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f82dc06ae70 0x7f82dc06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:11.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.536+0000 7f82ecff9700 1 -- 172.21.15.78:0/2575760923 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f82e808a730 con 0x7f82f01554e0 2023-10-14T11:58:11.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.536+0000 7f82effff700 1 --2- 172.21.15.78:0/2575760923 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f82dc06ae70 0x7f82dc06d330 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f82e4006b20 tx=0x7f82e4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:11.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.540+0000 7f82ecff9700 1 -- 172.21.15.78:0/2575760923 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f82e8055ea0 con 0x7f82f01554e0 2023-10-14T11:58:11.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.735+0000 7f82f65e1700 1 -- 172.21.15.78:0/2575760923 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82f0157e90 con 0x7f82f01554e0 2023-10-14T11:58:11.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.736+0000 7f82ecff9700 1 -- 172.21.15.78:0/2575760923 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12826 (secure 0 0 0) 0x7f82f0157e90 con 0x7f82f01554e0 2023-10-14T11:58:11.738 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:11.739 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":17,"fsid":"4c5dc134-3092-406a-a1e1-13e6d9e926e8","created":"2023-10-14T11:57:57.363750+0000","modified":"2023-10-14T11:58:11.372391+0000","last_up_change":"2023-10-14T11:58:06.982472+0000","last_in_change":"2023-10-14T11:57:59.821373+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T11:58:09.001200+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"625a5fc0-9474-4eae-b80e-14c27a74ad19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6801","nonce":3884969780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6803","nonce":3884969780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6807","nonce":3884969780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6805","nonce":3884969780}]},"public_addr":"172.21.15.78:6801/3884969780","cluster_addr":"172.21.15.78:6803/3884969780","heartbeat_back_addr":"172.21.15.78:6807/3884969780","heartbeat_front_addr":"172.21.15.78:6805/3884969780","state":["exists","up"]},{"osd":1,"uuid":"49d63db1-2981-49ab-9c40-c50fc744b982","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6809","nonce":2176928406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6811","nonce":2176928406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6815","nonce":2176928406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6813","nonce":2176928406}]},"public_addr":"172.21.15.78:6809/2176928406","cluster_addr":"172.21.15.78:6811/2176928406","heartbeat_back_addr":"172.21.15.78:6815/2176928406","heartbeat_front_addr":"172.21.15.78:6813/2176928406","state":["exists","up"]},{"osd":2,"uuid":"1d7d0b29-cbd1-4717-9fe6-765a8d4998b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6825","nonce":3647044577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6827","nonce":3647044577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6831","nonce":3647044577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6829","nonce":3647044577}]},"public_addr":"172.21.15.78:6825/3647044577","cluster_addr":"172.21.15.78:6827/3647044577","heartbeat_back_addr":"172.21.15.78:6831/3647044577","heartbeat_front_addr":"172.21.15.78:6829/3647044577","state":["exists","up"]},{"osd":3,"uuid":"2f75ded1-e9e2-40e7-99ae-45e1e425e1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6817","nonce":3584112288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6819","nonce":3584112288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6823","nonce":3584112288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6821","nonce":3584112288}]},"public_addr":"172.21.15.78:6817/3584112288","cluster_addr":"172.21.15.78:6819/3584112288","heartbeat_back_addr":"172.21.15.78:6823/3584112288","heartbeat_front_addr":"172.21.15.78:6821/3584112288","state":["exists","up"]},{"osd":4,"uuid":"5130b432-8358-46cb-8a31-3064c976f6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6801","nonce":3732756301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6803","nonce":3732756301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6807","nonce":3732756301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6805","nonce":3732756301}]},"public_addr":"172.21.15.195:6801/3732756301","cluster_addr":"172.21.15.195:6803/3732756301","heartbeat_back_addr":"172.21.15.195:6807/3732756301","heartbeat_front_addr":"172.21.15.195:6805/3732756301","state":["exists","up"]},{"osd":5,"uuid":"60e234a9-a506-4e40-9c19-49a58f3cf28c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6825","nonce":2857706411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6827","nonce":2857706411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6831","nonce":2857706411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6829","nonce":2857706411}]},"public_addr":"172.21.15.195:6825/2857706411","cluster_addr":"172.21.15.195:6827/2857706411","heartbeat_back_addr":"172.21.15.195:6831/2857706411","heartbeat_front_addr":"172.21.15.195:6829/2857706411","state":["exists","up"]},{"osd":6,"uuid":"df7d2dd6-26b6-4eb1-96e5-94d1f3955b33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6810","nonce":72554374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6813","nonce":72554374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6820","nonce":72554374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6817","nonce":72554374}]},"public_addr":"172.21.15.195:6810/72554374","cluster_addr":"172.21.15.195:6813/72554374","heartbeat_back_addr":"172.21.15.195:6820/72554374","heartbeat_front_addr":"172.21.15.195:6817/72554374","state":["exists","up"]},{"osd":7,"uuid":"47a9d4de-f56d-4aa4-9900-c4a9cf9b2f97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6809","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6811","nonce":482472979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6816","nonce":482472979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6823","nonce":482472979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6821","nonce":482472979}]},"public_addr":"172.21.15.195:6811/482472979","cluster_addr":"172.21.15.195:6816/482472979","heartbeat_back_addr":"172.21.15.195:6823/482472979","heartbeat_front_addr":"172.21.15.195:6821/482472979","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.754003+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.005994+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.043603+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.024506+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.062585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.033767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.994825+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.038300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T11:58:11.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.738+0000 7f82f65e1700 1 -- 172.21.15.78:0/2575760923 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f82dc06ae70 msgr2=0x7f82dc06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:11.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.738+0000 7f82f65e1700 1 --2- 172.21.15.78:0/2575760923 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f82dc06ae70 0x7f82dc06d330 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f82e4006b20 tx=0x7f82e4008040 comp rx=0 tx=0).stop 2023-10-14T11:58:11.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.738+0000 7f82f65e1700 1 -- 172.21.15.78:0/2575760923 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82f01554e0 msgr2=0x7f82f0134360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:11.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.738+0000 7f82f65e1700 1 --2- 172.21.15.78:0/2575760923 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82f01554e0 0x7f82f0134360 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f82f00ad290 tx=0x7f82e8004ad0 comp rx=0 tx=0).stop 2023-10-14T11:58:11.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.738+0000 7f82f65e1700 1 -- 172.21.15.78:0/2575760923 shutdown_connections 2023-10-14T11:58:11.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.738+0000 7f82f65e1700 1 --2- 172.21.15.78:0/2575760923 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82f01554e0 0x7f82f0134360 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:11.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.738+0000 7f82f65e1700 1 --2- 172.21.15.78:0/2575760923 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f82dc06ae70 0x7f82dc06d330 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:11.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.738+0000 7f82f65e1700 1 --2- 172.21.15.78:0/2575760923 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f82f006ab40 0x7f82f0138580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:11.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.738+0000 7f82f65e1700 1 --2- 172.21.15.78:0/2575760923 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f82f00c0350 0x7f82f00c4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:11.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.738+0000 7f82f65e1700 1 -- 172.21.15.78:0/2575760923 >> 172.21.15.78:0/2575760923 conn(0x7f82f00c3ac0 msgr2=0x7f82f01539a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:11.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.738+0000 7f82f65e1700 1 -- 172.21.15.78:0/2575760923 shutdown_connections 2023-10-14T11:58:11.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.738+0000 7f82f65e1700 1 -- 172.21.15.78:0/2575760923 wait complete. 2023-10-14T11:58:11.752 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T11:58:09.001200+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T11:58:11.753 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T11:58:11.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.881+0000 7f7e73bed700 1 Processor -- start 2023-10-14T11:58:11.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.881+0000 7f7e73bed700 1 -- start start 2023-10-14T11:58:11.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.882+0000 7f7e73bed700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7e6c0c35e0 0x7f7e6c1535d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:11.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.882+0000 7f7e73bed700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e6c154c20 0x7f7e6c155070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:11.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.882+0000 7f7e73bed700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e6c0c26d0 con 0x7f7e6c14ed10 2023-10-14T11:58:11.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.882+0000 7f7e73bed700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e6c0a8520 con 0x7f7e6c0c35e0 2023-10-14T11:58:11.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.882+0000 7f7e73bed700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6c0c29b0 con 0x7f7e6c154c20 2023-10-14T11:58:11.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.882+0000 7f7e71188700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7e6c0c35e0 0x7f7e6c1535d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:11.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.882+0000 7f7e71989700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f7e6c14ed10 0x7f7e6c152ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49190/0 (socket says 172.21.15.78:49190) 2023-10-14T11:58:11.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.882+0000 7f7e71989700 1 -- 172.21.15.78:0/4096069188 learned_addr learned my addr 172.21.15.78:0/4096069188 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:11.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.882+0000 7f7e7218a700 1 --2- 172.21.15.78:0/4096069188 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e6c154c20 0x7f7e6c155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:11.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.882+0000 7f7e71188700 1 -- 172.21.15.78:0/4096069188 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e6c154c20 msgr2=0x7f7e6c155070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:11.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.882+0000 7f7e71188700 1 --2- 172.21.15.78:0/4096069188 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e6c154c20 0x7f7e6c155070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:11.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.882+0000 7f7e71188700 1 -- 172.21.15.78:0/4096069188 >> v1:172.21.15.78:6789/0 conn(0x7f7e6c14ed10 legacy=0x7f7e6c152ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:11.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.882+0000 7f7e70987700 1 -- 172.21.15.78:0/4096069188 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1828859822 0 0) 0x7f7e6c0c26d0 con 0x7f7e6c14ed10 2023-10-14T11:58:11.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.882+0000 7f7e71188700 1 -- 172.21.15.78:0/4096069188 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e6c0c2c90 con 0x7f7e6c0c35e0 2023-10-14T11:58:11.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.882+0000 7f7e71188700 1 --2- 172.21.15.78:0/4096069188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e6c0c35e0 0x7f7e6c1535d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c009840 tx=0x7f7e5c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d96edd6e155ebb51 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:11.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.882+0000 7f7e70987700 1 -- 172.21.15.78:0/4096069188 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e5c00f8a0 con 0x7f7e6c0c35e0 2023-10-14T11:58:11.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.882+0000 7f7e70987700 1 -- 172.21.15.78:0/4096069188 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e5c00fa00 con 0x7f7e6c0c35e0 2023-10-14T11:58:11.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.882+0000 7f7e70987700 1 -- 172.21.15.78:0/4096069188 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e5c00fd90 con 0x7f7e6c0c35e0 2023-10-14T11:58:11.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.883+0000 7f7e73bed700 1 -- 172.21.15.78:0/4096069188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e6c0c35e0 msgr2=0x7f7e6c1535d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:11.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.883+0000 7f7e73bed700 1 --2- 172.21.15.78:0/4096069188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e6c0c35e0 0x7f7e6c1535d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c009840 tx=0x7f7e5c009b50 comp rx=0 tx=0).stop 2023-10-14T11:58:11.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.883+0000 7f7e73bed700 1 -- 172.21.15.78:0/4096069188 shutdown_connections 2023-10-14T11:58:11.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.883+0000 7f7e73bed700 1 --2- 172.21.15.78:0/4096069188 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e6c154c20 0x7f7e6c155070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:11.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.883+0000 7f7e73bed700 1 --2- 172.21.15.78:0/4096069188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e6c0c35e0 0x7f7e6c1535d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:11.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.883+0000 7f7e73bed700 1 -- 172.21.15.78:0/4096069188 >> 172.21.15.78:0/4096069188 conn(0x7f7e6c0c3a50 msgr2=0x7f7e6c0c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:11.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.883+0000 7f7e73bed700 1 -- 172.21.15.78:0/4096069188 shutdown_connections 2023-10-14T11:58:11.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.883+0000 7f7e73bed700 1 -- 172.21.15.78:0/4096069188 wait complete. 2023-10-14T11:58:11.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.884+0000 7f7e73bed700 1 Processor -- start 2023-10-14T11:58:11.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.885+0000 7f7e73bed700 1 -- start start 2023-10-14T11:58:11.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.885+0000 7f7e73bed700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e6c14ed10 0x7f7e6c149dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:11.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.885+0000 7f7e73bed700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e6c154c20 0x7f7e6c13dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:11.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.885+0000 7f7e73bed700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e6c1e8790 0x7f7e6c0c1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:11.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.885+0000 7f7e73bed700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6c13e220 con 0x7f7e6c14ed10 2023-10-14T11:58:11.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.885+0000 7f7e73bed700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6c1e5220 con 0x7f7e6c154c20 2023-10-14T11:58:11.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.885+0000 7f7e73bed700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6c1e8c20 con 0x7f7e6c1e8790 2023-10-14T11:58:11.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.885+0000 7f7e71989700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e6c14ed10 0x7f7e6c149dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:11.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.886+0000 7f7e71989700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e6c14ed10 0x7f7e6c149dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44320/0 (socket says 172.21.15.78:44320) 2023-10-14T11:58:11.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.886+0000 7f7e71989700 1 -- 172.21.15.78:0/3241555365 learned_addr learned my addr 172.21.15.78:0/3241555365 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:11.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.886+0000 7f7e7218a700 1 --2- 172.21.15.78:0/3241555365 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e6c1e8790 0x7f7e6c0c1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:11.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.886+0000 7f7e71188700 1 --2- 172.21.15.78:0/3241555365 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e6c154c20 0x7f7e6c13dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:11.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.886+0000 7f7e71989700 1 -- 172.21.15.78:0/3241555365 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e6c1e8790 msgr2=0x7f7e6c0c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:11.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.886+0000 7f7e71989700 1 --2- 172.21.15.78:0/3241555365 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e6c1e8790 0x7f7e6c0c1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:11.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.886+0000 7f7e71989700 1 -- 172.21.15.78:0/3241555365 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e6c154c20 msgr2=0x7f7e6c13dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:11.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.886+0000 7f7e71989700 1 --2- 172.21.15.78:0/3241555365 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e6c154c20 0x7f7e6c13dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:11.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.886+0000 7f7e71989700 1 -- 172.21.15.78:0/3241555365 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e5c0094a0 con 0x7f7e6c14ed10 2023-10-14T11:58:11.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.886+0000 7f7e71188700 1 --2- 172.21.15.78:0/3241555365 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e6c154c20 0x7f7e6c13dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:11.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.886+0000 7f7e71989700 1 --2- 172.21.15.78:0/3241555365 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e6c14ed10 0x7f7e6c149dd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7e680011e0 tx=0x7f7e6800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:11.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.886+0000 7f7e627fc700 1 -- 172.21.15.78:0/3241555365 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e68004c80 con 0x7f7e6c14ed10 2023-10-14T11:58:11.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.886+0000 7f7e627fc700 1 -- 172.21.15.78:0/3241555365 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e68004de0 con 0x7f7e6c14ed10 2023-10-14T11:58:11.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.886+0000 7f7e73bed700 1 -- 172.21.15.78:0/3241555365 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e6c1e8f00 con 0x7f7e6c14ed10 2023-10-14T11:58:11.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.886+0000 7f7e627fc700 1 -- 172.21.15.78:0/3241555365 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e68007310 con 0x7f7e6c14ed10 2023-10-14T11:58:11.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.886+0000 7f7e73bed700 1 -- 172.21.15.78:0/3241555365 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e6c1e94c0 con 0x7f7e6c14ed10 2023-10-14T11:58:11.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.887+0000 7f7e73bed700 1 -- 172.21.15.78:0/3241555365 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e6c074b80 con 0x7f7e6c14ed10 2023-10-14T11:58:11.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.887+0000 7f7e627fc700 1 -- 172.21.15.78:0/3241555365 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7e68007470 con 0x7f7e6c14ed10 2023-10-14T11:58:11.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.888+0000 7f7e627fc700 1 --2- 172.21.15.78:0/3241555365 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7e5006af00 0x7f7e5006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:11.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.888+0000 7f7e627fc700 1 -- 172.21.15.78:0/3241555365 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7e68088ed0 con 0x7f7e6c14ed10 2023-10-14T11:58:11.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.888+0000 7f7e71188700 1 --2- 172.21.15.78:0/3241555365 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7e5006af00 0x7f7e5006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:11.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.888+0000 7f7e71188700 1 --2- 172.21.15.78:0/3241555365 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7e5006af00 0x7f7e5006d3c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c000c00 tx=0x7f7e5c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:11.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:11.892+0000 7f7e627fc700 1 -- 172.21.15.78:0/3241555365 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7e68054360 con 0x7f7e6c14ed10 2023-10-14T11:58:12.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.080+0000 7f7e73bed700 1 -- 172.21.15.78:0/3241555365 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7e6c06b760 con 0x7f7e6c14ed10 2023-10-14T11:58:12.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.080+0000 7f7e627fc700 1 -- 172.21.15.78:0/3241555365 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7e68057980 con 0x7f7e6c14ed10 2023-10-14T11:58:12.083 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-10-14T11:58:12.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.082+0000 7f7e73bed700 1 -- 172.21.15.78:0/3241555365 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7e5006af00 msgr2=0x7f7e5006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:12.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.082+0000 7f7e73bed700 1 --2- 172.21.15.78:0/3241555365 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7e5006af00 0x7f7e5006d3c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c000c00 tx=0x7f7e5c008030 comp rx=0 tx=0).stop 2023-10-14T11:58:12.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.082+0000 7f7e73bed700 1 -- 172.21.15.78:0/3241555365 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e6c14ed10 msgr2=0x7f7e6c149dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:12.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.082+0000 7f7e73bed700 1 --2- 172.21.15.78:0/3241555365 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e6c14ed10 0x7f7e6c149dd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7e680011e0 tx=0x7f7e6800ca30 comp rx=0 tx=0).stop 2023-10-14T11:58:12.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.082+0000 7f7e73bed700 1 -- 172.21.15.78:0/3241555365 shutdown_connections 2023-10-14T11:58:12.086 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.082+0000 7f7e73bed700 1 --2- 172.21.15.78:0/3241555365 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e6c1e8790 0x7f7e6c0c1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:12.086 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.082+0000 7f7e73bed700 1 --2- 172.21.15.78:0/3241555365 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e6c14ed10 0x7f7e6c149dd0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:12.086 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.082+0000 7f7e73bed700 1 --2- 172.21.15.78:0/3241555365 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7e5006af00 0x7f7e5006d3c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:12.086 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.082+0000 7f7e73bed700 1 --2- 172.21.15.78:0/3241555365 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e6c154c20 0x7f7e6c13dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:12.087 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.083+0000 7f7e73bed700 1 -- 172.21.15.78:0/3241555365 >> 172.21.15.78:0/3241555365 conn(0x7f7e6c0c3a50 msgr2=0x7f7e6c14bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:12.087 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.083+0000 7f7e73bed700 1 -- 172.21.15.78:0/3241555365 shutdown_connections 2023-10-14T11:58:12.087 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.083+0000 7f7e73bed700 1 -- 172.21.15.78:0/3241555365 wait complete. 2023-10-14T11:58:12.096 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-14T11:58:12.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.215+0000 7fd2e6ebf700 1 Processor -- start 2023-10-14T11:58:12.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.215+0000 7fd2e6ebf700 1 -- start start 2023-10-14T11:58:12.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.215+0000 7fd2e6ebf700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd2e00c44c0 0x7fd2e00c7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:12.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.215+0000 7fd2e6ebf700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd2e00c72a0 0x7fd2e00c86e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:12.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.215+0000 7fd2e6ebf700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e0156b20 con 0x7fd2e00c8c20 2023-10-14T11:58:12.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.215+0000 7fd2e6ebf700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2e0155f10 con 0x7fd2e00c44c0 2023-10-14T11:58:12.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.216+0000 7fd2e6ebf700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e01561f0 con 0x7fd2e00c72a0 2023-10-14T11:58:12.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.216+0000 7fd2e4c5b700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd2e00c44c0 0x7fd2e00c7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:12.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.216+0000 7fd2e545c700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd2e00c8c20 0x7fd2e00c2c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49208/0 (socket says 172.21.15.78:49208) 2023-10-14T11:58:12.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.216+0000 7fd2e545c700 1 -- 172.21.15.78:0/2856569140 learned_addr learned my addr 172.21.15.78:0/2856569140 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:12.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.216+0000 7fd2dffff700 1 --2- 172.21.15.78:0/2856569140 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd2e00c72a0 0x7fd2e00c86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:12.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.216+0000 7fd2e4c5b700 1 -- 172.21.15.78:0/2856569140 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd2e00c72a0 msgr2=0x7fd2e00c86e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:12.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.216+0000 7fd2e4c5b700 1 --2- 172.21.15.78:0/2856569140 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd2e00c72a0 0x7fd2e00c86e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:12.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.216+0000 7fd2e4c5b700 1 -- 172.21.15.78:0/2856569140 >> v1:172.21.15.78:6789/0 conn(0x7fd2e00c8c20 legacy=0x7fd2e00c2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:12.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.216+0000 7fd2e4c5b700 1 -- 172.21.15.78:0/2856569140 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e01564d0 con 0x7fd2e00c44c0 2023-10-14T11:58:12.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.216+0000 7fd2df7fe700 1 -- 172.21.15.78:0/2856569140 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430373085 0 0) 0x7fd2e0156b20 con 0x7fd2e00c8c20 2023-10-14T11:58:12.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.216+0000 7fd2e4c5b700 1 --2- 172.21.15.78:0/2856569140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2e00c44c0 0x7fd2e00c7a00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd2d4009840 tx=0x7fd2d4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b08ad07d10d2fd69 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:12.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.216+0000 7fd2df7fe700 1 -- 172.21.15.78:0/2856569140 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d4011830 con 0x7fd2e00c44c0 2023-10-14T11:58:12.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.216+0000 7fd2df7fe700 1 -- 172.21.15.78:0/2856569140 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d4011990 con 0x7fd2e00c44c0 2023-10-14T11:58:12.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.216+0000 7fd2df7fe700 1 -- 172.21.15.78:0/2856569140 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d4011d20 con 0x7fd2e00c44c0 2023-10-14T11:58:12.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.217+0000 7fd2e6ebf700 1 -- 172.21.15.78:0/2856569140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2e00c44c0 msgr2=0x7fd2e00c7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:12.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.217+0000 7fd2e6ebf700 1 --2- 172.21.15.78:0/2856569140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2e00c44c0 0x7fd2e00c7a00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd2d4009840 tx=0x7fd2d4009b50 comp rx=0 tx=0).stop 2023-10-14T11:58:12.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.217+0000 7fd2e6ebf700 1 -- 172.21.15.78:0/2856569140 shutdown_connections 2023-10-14T11:58:12.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.217+0000 7fd2e6ebf700 1 --2- 172.21.15.78:0/2856569140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2e00c44c0 0x7fd2e00c7a00 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:12.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.217+0000 7fd2e6ebf700 1 --2- 172.21.15.78:0/2856569140 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd2e00c72a0 0x7fd2e00c86e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:12.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.217+0000 7fd2e6ebf700 1 -- 172.21.15.78:0/2856569140 >> 172.21.15.78:0/2856569140 conn(0x7fd2e00737d0 msgr2=0x7fd2e0073be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:12.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.217+0000 7fd2e6ebf700 1 -- 172.21.15.78:0/2856569140 shutdown_connections 2023-10-14T11:58:12.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.217+0000 7fd2e6ebf700 1 -- 172.21.15.78:0/2856569140 wait complete. 2023-10-14T11:58:12.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.218+0000 7fd2e6ebf700 1 Processor -- start 2023-10-14T11:58:12.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.218+0000 7fd2e6ebf700 1 -- start start 2023-10-14T11:58:12.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.219+0000 7fd2e6ebf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2e00c44c0 0x7fd2e00c1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:12.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.219+0000 7fd2e6ebf700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd2e00c72a0 0x7fd2e00ad630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:12.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.219+0000 7fd2e6ebf700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd2e00c8c20 0x7fd2e014fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:12.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.219+0000 7fd2e6ebf700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e00bcb30 con 0x7fd2e00c44c0 2023-10-14T11:58:12.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.219+0000 7fd2e6ebf700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e01e78e0 con 0x7fd2e00c72a0 2023-10-14T11:58:12.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.219+0000 7fd2e6ebf700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e01e7b80 con 0x7fd2e00c8c20 2023-10-14T11:58:12.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.219+0000 7fd2e4c5b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2e00c44c0 0x7fd2e00c1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:12.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.219+0000 7fd2e4c5b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2e00c44c0 0x7fd2e00c1670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44334/0 (socket says 172.21.15.78:44334) 2023-10-14T11:58:12.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.219+0000 7fd2e4c5b700 1 -- 172.21.15.78:0/1721099076 learned_addr learned my addr 172.21.15.78:0/1721099076 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:12.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.220+0000 7fd2e545c700 1 --2- 172.21.15.78:0/1721099076 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd2e00c8c20 0x7fd2e014fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:12.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.220+0000 7fd2dffff700 1 --2- 172.21.15.78:0/1721099076 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd2e00c72a0 0x7fd2e00ad630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:12.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.220+0000 7fd2e4c5b700 1 -- 172.21.15.78:0/1721099076 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd2e00c8c20 msgr2=0x7fd2e014fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:12.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.220+0000 7fd2e4c5b700 1 --2- 172.21.15.78:0/1721099076 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd2e00c8c20 0x7fd2e014fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:12.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.220+0000 7fd2e4c5b700 1 -- 172.21.15.78:0/1721099076 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd2e00c72a0 msgr2=0x7fd2e00ad630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:12.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.220+0000 7fd2e4c5b700 1 --2- 172.21.15.78:0/1721099076 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd2e00c72a0 0x7fd2e00ad630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:12.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.220+0000 7fd2e4c5b700 1 -- 172.21.15.78:0/1721099076 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d40094a0 con 0x7fd2e00c44c0 2023-10-14T11:58:12.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.220+0000 7fd2dffff700 1 --2- 172.21.15.78:0/1721099076 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd2e00c72a0 0x7fd2e00ad630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:12.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.220+0000 7fd2e4c5b700 1 --2- 172.21.15.78:0/1721099076 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2e00c44c0 0x7fd2e00c1670 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd2d40090b0 tx=0x7fd2d4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:12.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.220+0000 7fd2dd7fa700 1 -- 172.21.15.78:0/1721099076 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d40036a0 con 0x7fd2e00c44c0 2023-10-14T11:58:12.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.220+0000 7fd2dd7fa700 1 -- 172.21.15.78:0/1721099076 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d401f070 con 0x7fd2e00c44c0 2023-10-14T11:58:12.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.220+0000 7fd2e6ebf700 1 -- 172.21.15.78:0/1721099076 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2e01e7e60 con 0x7fd2e00c44c0 2023-10-14T11:58:12.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.220+0000 7fd2dd7fa700 1 -- 172.21.15.78:0/1721099076 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d4003a00 con 0x7fd2e00c44c0 2023-10-14T11:58:12.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.220+0000 7fd2e6ebf700 1 -- 172.21.15.78:0/1721099076 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2e00c1bb0 con 0x7fd2e00c44c0 2023-10-14T11:58:12.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.221+0000 7fd2e6ebf700 1 -- 172.21.15.78:0/1721099076 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2e0074b80 con 0x7fd2e00c44c0 2023-10-14T11:58:12.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.221+0000 7fd2dd7fa700 1 -- 172.21.15.78:0/1721099076 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd2d4003b60 con 0x7fd2e00c44c0 2023-10-14T11:58:12.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.222+0000 7fd2dd7fa700 1 --2- 172.21.15.78:0/1721099076 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd2c406af00 0x7fd2c406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:12.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.222+0000 7fd2dd7fa700 1 -- 172.21.15.78:0/1721099076 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd2d4021070 con 0x7fd2e00c44c0 2023-10-14T11:58:12.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.222+0000 7fd2dffff700 1 --2- 172.21.15.78:0/1721099076 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd2c406af00 0x7fd2c406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:12.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.222+0000 7fd2dffff700 1 --2- 172.21.15.78:0/1721099076 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd2c406af00 0x7fd2c406d3c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd2e0070480 tx=0x7fd2cc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:12.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.226+0000 7fd2dd7fa700 1 -- 172.21.15.78:0/1721099076 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd2d4055ff0 con 0x7fd2e00c44c0 2023-10-14T11:58:12.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.418+0000 7fd2e6ebf700 1 -- 172.21.15.78:0/1721099076 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd2e00c7080 con 0x7fd2e00c44c0 2023-10-14T11:58:12.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.419+0000 7fd2dd7fa700 1 -- 172.21.15.78:0/1721099076 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) v1 ==== 84+0+65 (secure 0 0 0) 0x7fd2d4015020 con 0x7fd2e00c44c0 2023-10-14T11:58:12.421 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:12.421 INFO:teuthology.orchestra.run.smithi078.stdout:[ 2023-10-14T11:58:12.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-14T11:58:12.422 INFO:teuthology.orchestra.run.smithi078.stdout: "poolnum": 1, 2023-10-14T11:58:12.422 INFO:teuthology.orchestra.run.smithi078.stdout: "poolname": ".mgr" 2023-10-14T11:58:12.422 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-14T11:58:12.423 INFO:teuthology.orchestra.run.smithi078.stdout:] 2023-10-14T11:58:12.424 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.421+0000 7fd2e6ebf700 1 -- 172.21.15.78:0/1721099076 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd2c406af00 msgr2=0x7fd2c406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:12.424 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.421+0000 7fd2e6ebf700 1 --2- 172.21.15.78:0/1721099076 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd2c406af00 0x7fd2c406d3c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd2e0070480 tx=0x7fd2cc008040 comp rx=0 tx=0).stop 2023-10-14T11:58:12.424 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.421+0000 7fd2e6ebf700 1 -- 172.21.15.78:0/1721099076 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2e00c44c0 msgr2=0x7fd2e00c1670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:12.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.421+0000 7fd2e6ebf700 1 --2- 172.21.15.78:0/1721099076 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2e00c44c0 0x7fd2e00c1670 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd2d40090b0 tx=0x7fd2d4004750 comp rx=0 tx=0).stop 2023-10-14T11:58:12.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.421+0000 7fd2e6ebf700 1 -- 172.21.15.78:0/1721099076 shutdown_connections 2023-10-14T11:58:12.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.421+0000 7fd2e6ebf700 1 --2- 172.21.15.78:0/1721099076 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd2e00c8c20 0x7fd2e014fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:12.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.421+0000 7fd2e6ebf700 1 --2- 172.21.15.78:0/1721099076 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2e00c44c0 0x7fd2e00c1670 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:12.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.421+0000 7fd2e6ebf700 1 --2- 172.21.15.78:0/1721099076 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd2c406af00 0x7fd2c406d3c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:12.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.421+0000 7fd2e6ebf700 1 --2- 172.21.15.78:0/1721099076 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd2e00c72a0 0x7fd2e00ad630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:12.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.421+0000 7fd2e6ebf700 1 -- 172.21.15.78:0/1721099076 >> 172.21.15.78:0/1721099076 conn(0x7fd2e00737d0 msgr2=0x7fd2e00a9140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:12.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.421+0000 7fd2e6ebf700 1 -- 172.21.15.78:0/1721099076 shutdown_connections 2023-10-14T11:58:12.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.421+0000 7fd2e6ebf700 1 -- 172.21.15.78:0/1721099076 wait complete. 2023-10-14T11:58:12.435 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-14T11:58:12.435 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-14T11:58:12.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.556+0000 7f0692c37700 1 Processor -- start 2023-10-14T11:58:12.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.556+0000 7f0692c37700 1 -- start start 2023-10-14T11:58:12.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.557+0000 7f0692c37700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f068c14feb0 0x7f068c06dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:12.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.557+0000 7f0692c37700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f068c06d920 0x7f068c0c10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:12.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.557+0000 7f0692c37700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f068c0c66e0 con 0x7f068c0c1610 2023-10-14T11:58:12.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.557+0000 7f0692c37700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f068c06e4e0 con 0x7f068c14feb0 2023-10-14T11:58:12.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.557+0000 7f0692c37700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f068c150290 con 0x7f068c06d920 2023-10-14T11:58:12.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.557+0000 7f06909d3700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f068c14feb0 0x7f068c06dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:12.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.557+0000 7f06909d3700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f068c14feb0 0x7f068c06dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44342/0 (socket says 172.21.15.78:44342) 2023-10-14T11:58:12.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.557+0000 7f06909d3700 1 -- 172.21.15.78:0/693960313 learned_addr learned my addr 172.21.15.78:0/693960313 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:12.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.557+0000 7f068bfff700 1 --2- 172.21.15.78:0/693960313 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f068c06d920 0x7f068c0c10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:12.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.557+0000 7f06909d3700 1 -- 172.21.15.78:0/693960313 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f068c06d920 msgr2=0x7f068c0c10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:12.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.557+0000 7f06909d3700 1 --2- 172.21.15.78:0/693960313 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f068c06d920 0x7f068c0c10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:12.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.557+0000 7f06909d3700 1 -- 172.21.15.78:0/693960313 >> v1:172.21.15.195:6789/0 conn(0x7f068c0c1610 legacy=0x7f068c071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:12.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.557+0000 7f06909d3700 1 -- 172.21.15.78:0/693960313 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f068c06e6f0 con 0x7f068c14feb0 2023-10-14T11:58:12.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.558+0000 7f06909d3700 1 --2- 172.21.15.78:0/693960313 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f068c14feb0 0x7f068c06dfa0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f06800098f0 tx=0x7f0680009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89e9e58b0e14d430 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:12.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.558+0000 7f068b7fe700 1 -- 172.21.15.78:0/693960313 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f068000b8a0 con 0x7f068c14feb0 2023-10-14T11:58:12.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.558+0000 7f068b7fe700 1 -- 172.21.15.78:0/693960313 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f068000ba00 con 0x7f068c14feb0 2023-10-14T11:58:12.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.558+0000 7f068b7fe700 1 -- 172.21.15.78:0/693960313 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f068000bd90 con 0x7f068c14feb0 2023-10-14T11:58:12.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.558+0000 7f0692c37700 1 -- 172.21.15.78:0/693960313 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f068c14feb0 msgr2=0x7f068c06dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:12.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.558+0000 7f0692c37700 1 --2- 172.21.15.78:0/693960313 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f068c14feb0 0x7f068c06dfa0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f06800098f0 tx=0x7f0680009c00 comp rx=0 tx=0).stop 2023-10-14T11:58:12.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.558+0000 7f0692c37700 1 -- 172.21.15.78:0/693960313 shutdown_connections 2023-10-14T11:58:12.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.558+0000 7f0692c37700 1 --2- 172.21.15.78:0/693960313 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f068c14feb0 0x7f068c06dfa0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:12.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.558+0000 7f0692c37700 1 --2- 172.21.15.78:0/693960313 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f068c06d920 0x7f068c0c10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:12.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.558+0000 7f0692c37700 1 -- 172.21.15.78:0/693960313 >> 172.21.15.78:0/693960313 conn(0x7f068c0742a0 msgr2=0x7f068c0a8490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:12.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.558+0000 7f0692c37700 1 -- 172.21.15.78:0/693960313 shutdown_connections 2023-10-14T11:58:12.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.558+0000 7f0692c37700 1 -- 172.21.15.78:0/693960313 wait complete. 2023-10-14T11:58:12.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.559+0000 7f0692c37700 1 Processor -- start 2023-10-14T11:58:12.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.560+0000 7f0692c37700 1 -- start start 2023-10-14T11:58:12.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.560+0000 7f0692c37700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f068c06d920 0x7f068c148570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:12.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.560+0000 7f0692c37700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f068c0c1610 0x7f068c1465c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:12.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.560+0000 7f0692c37700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f068c14feb0 0x7f068c146b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:12.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.560+0000 7f0692c37700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068c0c0da0 con 0x7f068c0c1610 2023-10-14T11:58:12.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.560+0000 7f0692c37700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068c147040 con 0x7f068c14feb0 2023-10-14T11:58:12.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.560+0000 7f0692c37700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f068c155a10 con 0x7f068c06d920 2023-10-14T11:58:12.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.560+0000 7f068bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f068c0c1610 0x7f068c1465c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:12.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.560+0000 7f068bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f068c0c1610 0x7f068c1465c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44346/0 (socket says 172.21.15.78:44346) 2023-10-14T11:58:12.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.560+0000 7f068bfff700 1 -- 172.21.15.78:0/727734715 learned_addr learned my addr 172.21.15.78:0/727734715 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:12.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.561+0000 7f06909d3700 1 --2- 172.21.15.78:0/727734715 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f068c06d920 0x7f068c148570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:12.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.561+0000 7f06911d4700 1 --2- 172.21.15.78:0/727734715 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f068c14feb0 0x7f068c146b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:12.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.561+0000 7f068bfff700 1 -- 172.21.15.78:0/727734715 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f068c06d920 msgr2=0x7f068c148570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:12.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.561+0000 7f068bfff700 1 --2- 172.21.15.78:0/727734715 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f068c06d920 0x7f068c148570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:12.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.561+0000 7f068bfff700 1 -- 172.21.15.78:0/727734715 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f068c14feb0 msgr2=0x7f068c146b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:12.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.561+0000 7f068bfff700 1 --2- 172.21.15.78:0/727734715 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f068c14feb0 0x7f068c146b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:12.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.561+0000 7f068bfff700 1 -- 172.21.15.78:0/727734715 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0680009550 con 0x7f068c0c1610 2023-10-14T11:58:12.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.561+0000 7f068bfff700 1 --2- 172.21.15.78:0/727734715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f068c0c1610 0x7f068c1465c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f067800d800 tx=0x7f067800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:12.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.561+0000 7f06897fa700 1 -- 172.21.15.78:0/727734715 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06780125f0 con 0x7f068c0c1610 2023-10-14T11:58:12.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.561+0000 7f06897fa700 1 -- 172.21.15.78:0/727734715 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0678012750 con 0x7f068c0c1610 2023-10-14T11:58:12.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.561+0000 7f0692c37700 1 -- 172.21.15.78:0/727734715 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f068c155cf0 con 0x7f068c0c1610 2023-10-14T11:58:12.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.561+0000 7f06897fa700 1 -- 172.21.15.78:0/727734715 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0678012b00 con 0x7f068c0c1610 2023-10-14T11:58:12.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.561+0000 7f0692c37700 1 -- 172.21.15.78:0/727734715 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f068c1562b0 con 0x7f068c0c1610 2023-10-14T11:58:12.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.562+0000 7f0692c37700 1 -- 172.21.15.78:0/727734715 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f068c074b80 con 0x7f068c0c1610 2023-10-14T11:58:12.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.563+0000 7f06897fa700 1 -- 172.21.15.78:0/727734715 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0678012c60 con 0x7f068c0c1610 2023-10-14T11:58:12.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.563+0000 7f06897fa700 1 --2- 172.21.15.78:0/727734715 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f067006af00 0x7f067006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:12.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.563+0000 7f06909d3700 1 --2- 172.21.15.78:0/727734715 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f067006af00 0x7f067006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:12.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.563+0000 7f06897fa700 1 -- 172.21.15.78:0/727734715 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0678089330 con 0x7f068c0c1610 2023-10-14T11:58:12.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.564+0000 7f06909d3700 1 --2- 172.21.15.78:0/727734715 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f067006af00 0x7f067006d3c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0680009c40 tx=0x7f0680008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:12.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.568+0000 7f06897fa700 1 -- 172.21.15.78:0/727734715 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0678054730 con 0x7f068c0c1610 2023-10-14T11:58:12.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:12.760+0000 7f0692c37700 1 -- 172.21.15.78:0/727734715 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f068c072520 con 0x7f068c0c1610 2023-10-14T11:58:13.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.123+0000 7f06897fa700 1 -- 172.21.15.78:0/727734715 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0678057d50 con 0x7f068c0c1610 2023-10-14T11:58:13.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.125+0000 7f0692c37700 1 -- 172.21.15.78:0/727734715 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f067006af00 msgr2=0x7f067006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:13.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.125+0000 7f0692c37700 1 --2- 172.21.15.78:0/727734715 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f067006af00 0x7f067006d3c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0680009c40 tx=0x7f0680008030 comp rx=0 tx=0).stop 2023-10-14T11:58:13.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.125+0000 7f0692c37700 1 -- 172.21.15.78:0/727734715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f068c0c1610 msgr2=0x7f068c1465c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:13.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.125+0000 7f0692c37700 1 --2- 172.21.15.78:0/727734715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f068c0c1610 0x7f068c1465c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f067800d800 tx=0x7f067800db10 comp rx=0 tx=0).stop 2023-10-14T11:58:13.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.125+0000 7f0692c37700 1 -- 172.21.15.78:0/727734715 shutdown_connections 2023-10-14T11:58:13.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.126+0000 7f0692c37700 1 --2- 172.21.15.78:0/727734715 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f067006af00 0x7f067006d3c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:13.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.126+0000 7f0692c37700 1 --2- 172.21.15.78:0/727734715 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f068c14feb0 0x7f068c146b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:13.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.126+0000 7f0692c37700 1 --2- 172.21.15.78:0/727734715 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f068c06d920 0x7f068c148570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:13.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.126+0000 7f0692c37700 1 --2- 172.21.15.78:0/727734715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f068c0c1610 0x7f068c1465c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:13.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.126+0000 7f0692c37700 1 -- 172.21.15.78:0/727734715 >> 172.21.15.78:0/727734715 conn(0x7f068c0742a0 msgr2=0x7f068c14be30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:13.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.126+0000 7f0692c37700 1 -- 172.21.15.78:0/727734715 shutdown_connections 2023-10-14T11:58:13.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.126+0000 7f0692c37700 1 -- 172.21.15.78:0/727734715 wait complete. 2023-10-14T11:58:13.130 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' created 2023-10-14T11:58:13.143 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-14T11:58:13.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.269+0000 7f8b29578700 1 Processor -- start 2023-10-14T11:58:13.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.269+0000 7f8b29578700 1 -- start start 2023-10-14T11:58:13.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.270+0000 7f8b29578700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8b24155070 0x7f8b2415e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:13.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.270+0000 7f8b29578700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b241537b0 con 0x7f8b241533d0 2023-10-14T11:58:13.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.270+0000 7f8b29578700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b240c7c50 con 0x7f8b240a70c0 2023-10-14T11:58:13.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.270+0000 7f8b29578700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b240c6fb0 con 0x7f8b24155070 2023-10-14T11:58:13.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.270+0000 7f8b22ffd700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f8b241533d0 0x7f8b240c8680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49230/0 (socket says 172.21.15.78:49230) 2023-10-14T11:58:13.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.270+0000 7f8b22ffd700 1 -- 172.21.15.78:0/3165881061 learned_addr learned my addr 172.21.15.78:0/3165881061 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:13.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.270+0000 7f8b21ffb700 1 -- 172.21.15.78:0/3165881061 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1270212537 0 0) 0x7f8b241537b0 con 0x7f8b241533d0 2023-10-14T11:58:13.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.270+0000 7f8b21ffb700 1 -- 172.21.15.78:0/3165881061 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b0c0035b0 con 0x7f8b241533d0 2023-10-14T11:58:13.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.270+0000 7f8b237fe700 1 --2- 172.21.15.78:0/3165881061 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8b24155070 0x7f8b2415e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:13.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.270+0000 7f8b21ffb700 1 -- 172.21.15.78:0/3165881061 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2922405242 0 0) 0x7f8b0c0035b0 con 0x7f8b241533d0 2023-10-14T11:58:13.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.270+0000 7f8b21ffb700 1 -- 172.21.15.78:0/3165881061 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8b24155070 msgr2=0x7f8b2415e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:13.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.270+0000 7f8b21ffb700 1 --2- 172.21.15.78:0/3165881061 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8b24155070 0x7f8b2415e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:13.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.270+0000 7f8b21ffb700 1 -- 172.21.15.78:0/3165881061 >> v1:172.21.15.195:6789/0 conn(0x7f8b240a70c0 legacy=0x7f8b24154960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:13.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.270+0000 7f8b21ffb700 1 -- 172.21.15.78:0/3165881061 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b24153e80 con 0x7f8b241533d0 2023-10-14T11:58:13.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.270+0000 7f8b21ffb700 1 -- 172.21.15.78:0/3165881061 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441799917 0 0) 0x7f8b14001670 con 0x7f8b241533d0 2023-10-14T11:58:13.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.270+0000 7f8b21ffb700 1 -- 172.21.15.78:0/3165881061 >> v1:172.21.15.78:6789/0 conn(0x7f8b241533d0 legacy=0x7f8b240c8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:13.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.270+0000 7f8b21ffb700 1 --2- 172.21.15.78:0/3165881061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b0c003a30 0x7f8b0c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:13.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.270+0000 7f8b21ffb700 1 --2- 172.21.15.78:0/3165881061 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8b0c004850 0x7f8b0c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:13.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.270+0000 7f8b21ffb700 1 --2- 172.21.15.78:0/3165881061 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8b24155070 0x7f8b0c005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:13.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.271+0000 7f8b21ffb700 1 -- 172.21.15.78:0/3165881061 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b240c6fb0 con 0x7f8b0c003a30 2023-10-14T11:58:13.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.271+0000 7f8b21ffb700 1 -- 172.21.15.78:0/3165881061 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0c00bb70 con 0x7f8b0c004850 2023-10-14T11:58:13.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.271+0000 7f8b21ffb700 1 -- 172.21.15.78:0/3165881061 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0c00bd30 con 0x7f8b24155070 2023-10-14T11:58:13.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.271+0000 7f8b22ffd700 1 --2- 172.21.15.78:0/3165881061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b0c003a30 0x7f8b0c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:13.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.271+0000 7f8b237fe700 1 --2- 172.21.15.78:0/3165881061 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8b24155070 0x7f8b0c005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:13.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.271+0000 7f8b227fc700 1 --2- 172.21.15.78:0/3165881061 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8b0c004850 0x7f8b0c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:13.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.271+0000 7f8b22ffd700 1 -- 172.21.15.78:0/3165881061 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8b24155070 msgr2=0x7f8b0c005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:13.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.271+0000 7f8b22ffd700 1 --2- 172.21.15.78:0/3165881061 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8b24155070 0x7f8b0c005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:13.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.271+0000 7f8b22ffd700 1 -- 172.21.15.78:0/3165881061 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8b0c004850 msgr2=0x7f8b0c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:13.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.271+0000 7f8b22ffd700 1 --2- 172.21.15.78:0/3165881061 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8b0c004850 0x7f8b0c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:13.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.271+0000 7f8b22ffd700 1 -- 172.21.15.78:0/3165881061 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b0c0035b0 con 0x7f8b0c003a30 2023-10-14T11:58:13.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.271+0000 7f8b227fc700 1 --2- 172.21.15.78:0/3165881061 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8b0c004850 0x7f8b0c004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:13.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.271+0000 7f8b22ffd700 1 --2- 172.21.15.78:0/3165881061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b0c003a30 0x7f8b0c003e50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8b1400c2a0 tx=0x7f8b0c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16749b99a9552134 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:13.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.271+0000 7f8b21ffb700 1 -- 172.21.15.78:0/3165881061 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b14007c10 con 0x7f8b0c003a30 2023-10-14T11:58:13.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.271+0000 7f8b21ffb700 1 -- 172.21.15.78:0/3165881061 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b1400cdc0 con 0x7f8b0c003a30 2023-10-14T11:58:13.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.271+0000 7f8b21ffb700 1 -- 172.21.15.78:0/3165881061 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b14005ba0 con 0x7f8b0c003a30 2023-10-14T11:58:13.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.272+0000 7f8b29578700 1 -- 172.21.15.78:0/3165881061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b0c003a30 msgr2=0x7f8b0c003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:13.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.272+0000 7f8b29578700 1 --2- 172.21.15.78:0/3165881061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b0c003a30 0x7f8b0c003e50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8b1400c2a0 tx=0x7f8b0c005880 comp rx=0 tx=0).stop 2023-10-14T11:58:13.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.272+0000 7f8b237fe700 1 -- 172.21.15.78:0/3165881061 reap_dead start 2023-10-14T11:58:13.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.272+0000 7f8b29578700 1 -- 172.21.15.78:0/3165881061 shutdown_connections 2023-10-14T11:58:13.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.272+0000 7f8b29578700 1 -- 172.21.15.78:0/3165881061 >> 172.21.15.78:0/3165881061 conn(0x7f8b240737d0 msgr2=0x7f8b24073be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:13.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.272+0000 7f8b29578700 1 -- 172.21.15.78:0/3165881061 shutdown_connections 2023-10-14T11:58:13.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.272+0000 7f8b29578700 1 -- 172.21.15.78:0/3165881061 wait complete. 2023-10-14T11:58:13.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.273+0000 7f8b29578700 1 Processor -- start 2023-10-14T11:58:13.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.273+0000 7f8b29578700 1 -- start start 2023-10-14T11:58:13.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.274+0000 7f8b29578700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8b0c003a30 0x7f8b240ad5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:13.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.274+0000 7f8b29578700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8b240a9d00 0x7f8b240ba910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:13.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.274+0000 7f8b29578700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b240bae50 0x7f8b240a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:13.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.274+0000 7f8b29578700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b240c7f30 con 0x7f8b240bae50 2023-10-14T11:58:13.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.274+0000 7f8b29578700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b240c7a70 con 0x7f8b240a9d00 2023-10-14T11:58:13.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.274+0000 7f8b29578700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b240c4f50 con 0x7f8b0c003a30 2023-10-14T11:58:13.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.274+0000 7f8b237fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b240bae50 0x7f8b240a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:13.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.274+0000 7f8b237fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b240bae50 0x7f8b240a8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44370/0 (socket says 172.21.15.78:44370) 2023-10-14T11:58:13.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.274+0000 7f8b237fe700 1 -- 172.21.15.78:0/1753205871 learned_addr learned my addr 172.21.15.78:0/1753205871 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:13.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.274+0000 7f8b227fc700 1 --2- 172.21.15.78:0/1753205871 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8b240a9d00 0x7f8b240ba910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:13.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.274+0000 7f8b22ffd700 1 --2- 172.21.15.78:0/1753205871 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8b0c003a30 0x7f8b240ad5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:13.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.274+0000 7f8b237fe700 1 -- 172.21.15.78:0/1753205871 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8b0c003a30 msgr2=0x7f8b240ad5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:13.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.274+0000 7f8b237fe700 1 --2- 172.21.15.78:0/1753205871 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8b0c003a30 0x7f8b240ad5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:13.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.274+0000 7f8b237fe700 1 -- 172.21.15.78:0/1753205871 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8b240a9d00 msgr2=0x7f8b240ba910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:13.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.274+0000 7f8b237fe700 1 --2- 172.21.15.78:0/1753205871 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8b240a9d00 0x7f8b240ba910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:13.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.274+0000 7f8b237fe700 1 -- 172.21.15.78:0/1753205871 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b241ecf00 con 0x7f8b240bae50 2023-10-14T11:58:13.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.274+0000 7f8b227fc700 1 --2- 172.21.15.78:0/1753205871 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8b240a9d00 0x7f8b240ba910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:13.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.274+0000 7f8b22ffd700 1 --2- 172.21.15.78:0/1753205871 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8b0c003a30 0x7f8b240ad5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:13.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.274+0000 7f8b237fe700 1 --2- 172.21.15.78:0/1753205871 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b240bae50 0x7f8b240a8c60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c011c80 tx=0x7f8b1c00f490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:13.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.274+0000 7f8b0b7fe700 1 -- 172.21.15.78:0/1753205871 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b1c00e040 con 0x7f8b240bae50 2023-10-14T11:58:13.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.274+0000 7f8b0b7fe700 1 -- 172.21.15.78:0/1753205871 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b1c00f040 con 0x7f8b240bae50 2023-10-14T11:58:13.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.274+0000 7f8b29578700 1 -- 172.21.15.78:0/1753205871 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b241ed1e0 con 0x7f8b240bae50 2023-10-14T11:58:13.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.274+0000 7f8b0b7fe700 1 -- 172.21.15.78:0/1753205871 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b1c015430 con 0x7f8b240bae50 2023-10-14T11:58:13.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.274+0000 7f8b29578700 1 -- 172.21.15.78:0/1753205871 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b241ed7a0 con 0x7f8b240bae50 2023-10-14T11:58:13.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.275+0000 7f8b29578700 1 -- 172.21.15.78:0/1753205871 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b24074b80 con 0x7f8b240bae50 2023-10-14T11:58:13.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.276+0000 7f8b0b7fe700 1 -- 172.21.15.78:0/1753205871 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8b1c0155b0 con 0x7f8b240bae50 2023-10-14T11:58:13.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.276+0000 7f8b0b7fe700 1 --2- 172.21.15.78:0/1753205871 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8b1006af00 0x7f8b1006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:13.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.276+0000 7f8b22ffd700 1 --2- 172.21.15.78:0/1753205871 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8b1006af00 0x7f8b1006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:13.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.276+0000 7f8b0b7fe700 1 -- 172.21.15.78:0/1753205871 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f8b1c089b90 con 0x7f8b240bae50 2023-10-14T11:58:13.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.277+0000 7f8b22ffd700 1 --2- 172.21.15.78:0/1753205871 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8b1006af00 0x7f8b1006d3c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8b241556e0 tx=0x7f8b14007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:13.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.281+0000 7f8b0b7fe700 1 -- 172.21.15.78:0/1753205871 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8b1c054eb0 con 0x7f8b240bae50 2023-10-14T11:58:13.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:13.471+0000 7f8b29578700 1 -- 172.21.15.78:0/1753205871 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f8b240c70f0 con 0x7f8b240bae50 2023-10-14T11:58:14.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.127+0000 7f8b0b7fe700 1 -- 172.21.15.78:0/1753205871 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) v1 ==== 170+0+0 (secure 0 0 0) 0x7f8b1c0584d0 con 0x7f8b240bae50 2023-10-14T11:58:14.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.130+0000 7f8b29578700 1 -- 172.21.15.78:0/1753205871 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8b1006af00 msgr2=0x7f8b1006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:14.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.130+0000 7f8b29578700 1 --2- 172.21.15.78:0/1753205871 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8b1006af00 0x7f8b1006d3c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8b241556e0 tx=0x7f8b14007000 comp rx=0 tx=0).stop 2023-10-14T11:58:14.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.130+0000 7f8b29578700 1 -- 172.21.15.78:0/1753205871 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b240bae50 msgr2=0x7f8b240a8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:14.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.130+0000 7f8b29578700 1 --2- 172.21.15.78:0/1753205871 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b240bae50 0x7f8b240a8c60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c011c80 tx=0x7f8b1c00f490 comp rx=0 tx=0).stop 2023-10-14T11:58:14.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.130+0000 7f8b29578700 1 -- 172.21.15.78:0/1753205871 shutdown_connections 2023-10-14T11:58:14.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.130+0000 7f8b29578700 1 --2- 172.21.15.78:0/1753205871 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b240bae50 0x7f8b240a8c60 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:14.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.130+0000 7f8b29578700 1 --2- 172.21.15.78:0/1753205871 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8b0c003a30 0x7f8b240ad5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:14.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.130+0000 7f8b29578700 1 --2- 172.21.15.78:0/1753205871 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8b1006af00 0x7f8b1006d3c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:14.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.130+0000 7f8b29578700 1 --2- 172.21.15.78:0/1753205871 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8b240a9d00 0x7f8b240ba910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:14.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.130+0000 7f8b29578700 1 -- 172.21.15.78:0/1753205871 >> 172.21.15.78:0/1753205871 conn(0x7f8b240737d0 msgr2=0x7f8b2406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:14.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.131+0000 7f8b29578700 1 -- 172.21.15.78:0/1753205871 shutdown_connections 2023-10-14T11:58:14.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.131+0000 7f8b29578700 1 -- 172.21.15.78:0/1753205871 wait complete. 2023-10-14T11:58:14.139 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' created 2023-10-14T11:58:14.160 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-14T11:58:14.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.285+0000 7f9bd7636700 1 Processor -- start 2023-10-14T11:58:14.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.285+0000 7f9bd7636700 1 -- start start 2023-10-14T11:58:14.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.286+0000 7f9bd7636700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9bd01533d0 0x7f9bd00c8680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:14.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.286+0000 7f9bd7636700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9bd00c8bc0 0x7f9bd0154960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:14.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.286+0000 7f9bd7636700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bd01537b0 con 0x7f9bd00a70c0 2023-10-14T11:58:14.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.286+0000 7f9bd7636700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bd00c6fb0 con 0x7f9bd00c8bc0 2023-10-14T11:58:14.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.286+0000 7f9bd7636700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd00c7f30 con 0x7f9bd01533d0 2023-10-14T11:58:14.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.286+0000 7f9bd4bd1700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9bd00c8bc0 0x7f9bd0154960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:14.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.286+0000 7f9bd4bd1700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9bd00c8bc0 0x7f9bd0154960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44376/0 (socket says 172.21.15.78:44376) 2023-10-14T11:58:14.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.286+0000 7f9bd4bd1700 1 -- 172.21.15.78:0/823143772 learned_addr learned my addr 172.21.15.78:0/823143772 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:14.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.286+0000 7f9bd53d2700 1 --2- 172.21.15.78:0/823143772 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9bd01533d0 0x7f9bd00c8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:14.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.286+0000 7f9bd4bd1700 1 -- 172.21.15.78:0/823143772 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9bd01533d0 msgr2=0x7f9bd00c8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:14.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.286+0000 7f9bd4bd1700 1 --2- 172.21.15.78:0/823143772 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9bd01533d0 0x7f9bd00c8680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:14.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.286+0000 7f9bd4bd1700 1 -- 172.21.15.78:0/823143772 >> v1:172.21.15.195:6789/0 conn(0x7f9bd00a70c0 legacy=0x7f9bd0154ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:14.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.286+0000 7f9bd4bd1700 1 -- 172.21.15.78:0/823143772 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd00c7c50 con 0x7f9bd00c8bc0 2023-10-14T11:58:14.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.286+0000 7f9bd4bd1700 1 --2- 172.21.15.78:0/823143772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bd00c8bc0 0x7f9bd0154960 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9bc00098f0 tx=0x7f9bc0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11f7818bd8d66a8d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:14.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.286+0000 7f9bc7fff700 1 -- 172.21.15.78:0/823143772 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc0004230 con 0x7f9bd00c8bc0 2023-10-14T11:58:14.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.286+0000 7f9bc7fff700 1 -- 172.21.15.78:0/823143772 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bc0004390 con 0x7f9bd00c8bc0 2023-10-14T11:58:14.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.286+0000 7f9bc7fff700 1 -- 172.21.15.78:0/823143772 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc00036a0 con 0x7f9bd00c8bc0 2023-10-14T11:58:14.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.287+0000 7f9bd7636700 1 -- 172.21.15.78:0/823143772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bd00c8bc0 msgr2=0x7f9bd0154960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:14.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.287+0000 7f9bd7636700 1 --2- 172.21.15.78:0/823143772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bd00c8bc0 0x7f9bd0154960 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9bc00098f0 tx=0x7f9bc0009c00 comp rx=0 tx=0).stop 2023-10-14T11:58:14.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.287+0000 7f9bd7636700 1 -- 172.21.15.78:0/823143772 shutdown_connections 2023-10-14T11:58:14.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.287+0000 7f9bd7636700 1 --2- 172.21.15.78:0/823143772 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9bd01533d0 0x7f9bd00c8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:14.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.287+0000 7f9bd7636700 1 --2- 172.21.15.78:0/823143772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bd00c8bc0 0x7f9bd0154960 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:14.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.287+0000 7f9bd7636700 1 -- 172.21.15.78:0/823143772 >> 172.21.15.78:0/823143772 conn(0x7f9bd00737d0 msgr2=0x7f9bd0073be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:14.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.287+0000 7f9bd7636700 1 -- 172.21.15.78:0/823143772 shutdown_connections 2023-10-14T11:58:14.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.287+0000 7f9bd7636700 1 -- 172.21.15.78:0/823143772 wait complete. 2023-10-14T11:58:14.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.288+0000 7f9bd7636700 1 Processor -- start 2023-10-14T11:58:14.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.288+0000 7f9bd7636700 1 -- start start 2023-10-14T11:58:14.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.289+0000 7f9bd7636700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bd00a70c0 0x7f9bd01edd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:14.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.289+0000 7f9bd7636700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9bd00c8bc0 0x7f9bd006c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:14.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.289+0000 7f9bd7636700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9bd01533d0 0x7f9bd006cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:14.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.289+0000 7f9bd7636700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd00c7a70 con 0x7f9bd00a70c0 2023-10-14T11:58:14.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.289+0000 7f9bd7636700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd006c590 con 0x7f9bd01533d0 2023-10-14T11:58:14.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.289+0000 7f9bd7636700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd006d430 con 0x7f9bd00c8bc0 2023-10-14T11:58:14.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.289+0000 7f9bd53d2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bd00a70c0 0x7f9bd01edd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:14.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.289+0000 7f9bd53d2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bd00a70c0 0x7f9bd01edd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44380/0 (socket says 172.21.15.78:44380) 2023-10-14T11:58:14.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.289+0000 7f9bd53d2700 1 -- 172.21.15.78:0/3266887503 learned_addr learned my addr 172.21.15.78:0/3266887503 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:14.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.289+0000 7f9bd4bd1700 1 --2- 172.21.15.78:0/3266887503 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9bd00c8bc0 0x7f9bd006c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:14.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.289+0000 7f9bd5bd3700 1 --2- 172.21.15.78:0/3266887503 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9bd01533d0 0x7f9bd006cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:14.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.289+0000 7f9bd53d2700 1 -- 172.21.15.78:0/3266887503 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9bd00c8bc0 msgr2=0x7f9bd006c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:14.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.289+0000 7f9bd53d2700 1 --2- 172.21.15.78:0/3266887503 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9bd00c8bc0 0x7f9bd006c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:14.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.289+0000 7f9bd53d2700 1 -- 172.21.15.78:0/3266887503 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9bd01533d0 msgr2=0x7f9bd006cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:14.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.289+0000 7f9bd53d2700 1 --2- 172.21.15.78:0/3266887503 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9bd01533d0 0x7f9bd006cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:14.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.289+0000 7f9bd53d2700 1 -- 172.21.15.78:0/3266887503 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc0009550 con 0x7f9bd00a70c0 2023-10-14T11:58:14.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.289+0000 7f9bd53d2700 1 --2- 172.21.15.78:0/3266887503 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bd00a70c0 0x7f9bd01edd10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc00c820 tx=0x7f9bcc00cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:14.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.289+0000 7f9bc5ffb700 1 -- 172.21.15.78:0/3266887503 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc011800 con 0x7f9bd00a70c0 2023-10-14T11:58:14.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.289+0000 7f9bc5ffb700 1 -- 172.21.15.78:0/3266887503 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bcc011960 con 0x7f9bd00a70c0 2023-10-14T11:58:14.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.289+0000 7f9bd7636700 1 -- 172.21.15.78:0/3266887503 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bd01ee9e0 con 0x7f9bd00a70c0 2023-10-14T11:58:14.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.289+0000 7f9bc5ffb700 1 -- 172.21.15.78:0/3266887503 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc011cf0 con 0x7f9bd00a70c0 2023-10-14T11:58:14.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.290+0000 7f9bd7636700 1 -- 172.21.15.78:0/3266887503 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd01ee700 con 0x7f9bd00a70c0 2023-10-14T11:58:14.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.291+0000 7f9bd7636700 1 -- 172.21.15.78:0/3266887503 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bd0074b80 con 0x7f9bd00a70c0 2023-10-14T11:58:14.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.291+0000 7f9bc5ffb700 1 -- 172.21.15.78:0/3266887503 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f9bcc018360 con 0x7f9bd00a70c0 2023-10-14T11:58:14.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.291+0000 7f9bc5ffb700 1 --2- 172.21.15.78:0/3266887503 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f9bb406af00 0x7f9bb406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:14.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.292+0000 7f9bc5ffb700 1 -- 172.21.15.78:0/3266887503 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f9bcc0893d0 con 0x7f9bd00a70c0 2023-10-14T11:58:14.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.292+0000 7f9bd4bd1700 1 --2- 172.21.15.78:0/3266887503 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f9bb406af00 0x7f9bb406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:14.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.292+0000 7f9bd4bd1700 1 --2- 172.21.15.78:0/3266887503 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f9bb406af00 0x7f9bb406d3c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0009130 tx=0x7f9bc0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.296+0000 7f9bc5ffb700 1 -- 172.21.15.78:0/3266887503 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9bcc054570 con 0x7f9bd00a70c0 2023-10-14T11:58:14.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:14.533+0000 7f9bd7636700 1 -- 172.21.15.78:0/3266887503 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f9bd00c2260 con 0x7f9bd00a70c0 2023-10-14T11:58:15.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.150+0000 7f9bc5ffb700 1 -- 172.21.15.78:0/3266887503 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f9bcc057b90 con 0x7f9bd00a70c0 2023-10-14T11:58:15.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.158+0000 7f9bd7636700 1 -- 172.21.15.78:0/3266887503 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f9bb406af00 msgr2=0x7f9bb406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:15.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.158+0000 7f9bd7636700 1 --2- 172.21.15.78:0/3266887503 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f9bb406af00 0x7f9bb406d3c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0009130 tx=0x7f9bc0008030 comp rx=0 tx=0).stop 2023-10-14T11:58:15.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.158+0000 7f9bd7636700 1 -- 172.21.15.78:0/3266887503 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bd00a70c0 msgr2=0x7f9bd01edd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:15.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.158+0000 7f9bd7636700 1 --2- 172.21.15.78:0/3266887503 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bd00a70c0 0x7f9bd01edd10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc00c820 tx=0x7f9bcc00cbe0 comp rx=0 tx=0).stop 2023-10-14T11:58:15.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.158+0000 7f9bd7636700 1 -- 172.21.15.78:0/3266887503 shutdown_connections 2023-10-14T11:58:15.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.158+0000 7f9bd7636700 1 --2- 172.21.15.78:0/3266887503 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f9bb406af00 0x7f9bb406d3c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:15.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.158+0000 7f9bd7636700 1 --2- 172.21.15.78:0/3266887503 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9bd01533d0 0x7f9bd006cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:15.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.158+0000 7f9bd7636700 1 --2- 172.21.15.78:0/3266887503 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bd00a70c0 0x7f9bd01edd10 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc00c820 tx=0x7f9bcc00cbe0 comp rx=0 tx=0).stop 2023-10-14T11:58:15.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.158+0000 7f9bd7636700 1 --2- 172.21.15.78:0/3266887503 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9bd00c8bc0 0x7f9bd006c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:15.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.158+0000 7f9bd7636700 1 -- 172.21.15.78:0/3266887503 >> 172.21.15.78:0/3266887503 conn(0x7f9bd00737d0 msgr2=0x7f9bd00a8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:15.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.159+0000 7f9bd7636700 1 -- 172.21.15.78:0/3266887503 shutdown_connections 2023-10-14T11:58:15.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.160+0000 7f9bd7636700 1 -- 172.21.15.78:0/3266887503 wait complete. 2023-10-14T11:58:15.166 INFO:teuthology.orchestra.run.smithi078.stderr:new fs with metadata pool 2 and data pool 3 2023-10-14T11:58:15.182 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T11:58:15.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.301+0000 7f7901bee700 1 Processor -- start 2023-10-14T11:58:15.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.301+0000 7f7901bee700 1 -- start start 2023-10-14T11:58:15.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.302+0000 7f7901bee700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f78fc0c44c0 0x7f78fc0c7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:15.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.302+0000 7f7901bee700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78fc156b20 con 0x7f78fc0c72a0 2023-10-14T11:58:15.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.302+0000 7f7901bee700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78fc155f10 con 0x7f78fc0c2c90 2023-10-14T11:58:15.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.302+0000 7f7901bee700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78fc1561f0 con 0x7f78fc0c44c0 2023-10-14T11:58:15.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.302+0000 7f78fb7fe700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f78fc0c44c0 0x7f78fc0c7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:15.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.302+0000 7f78fb7fe700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f78fc0c44c0 0x7f78fc0c7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44402/0 (socket says 172.21.15.78:44402) 2023-10-14T11:58:15.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.302+0000 7f78faffd700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f78fc0c72a0 0x7f78fc0c86e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49278/0 (socket says 172.21.15.78:49278) 2023-10-14T11:58:15.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.302+0000 7f78fb7fe700 1 -- 172.21.15.78:0/3613911177 learned_addr learned my addr 172.21.15.78:0/3613911177 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:15.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.302+0000 7f78fb7fe700 1 -- 172.21.15.78:0/3613911177 >> v1:172.21.15.195:6789/0 conn(0x7f78fc0c2c90 legacy=0x7f78fc0c30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:15.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.302+0000 7f78fb7fe700 1 -- 172.21.15.78:0/3613911177 >> v1:172.21.15.78:6789/0 conn(0x7f78fc0c72a0 legacy=0x7f78fc0c86e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:15.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.302+0000 7f78fa7fc700 1 -- 172.21.15.78:0/3613911177 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4034933010 0 0) 0x7f78fc156b20 con 0x7f78fc0c72a0 2023-10-14T11:58:15.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.302+0000 7f78fb7fe700 1 -- 172.21.15.78:0/3613911177 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78fc1564d0 con 0x7f78fc0c44c0 2023-10-14T11:58:15.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.302+0000 7f78fb7fe700 1 --2- 172.21.15.78:0/3613911177 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78fc0c44c0 0x7f78fc0c7a00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f78e40098f0 tx=0x7f78e4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=168b785f62ea02e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:15.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.302+0000 7f78fa7fc700 1 -- 172.21.15.78:0/3613911177 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e4004230 con 0x7f78fc0c44c0 2023-10-14T11:58:15.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.303+0000 7f78fa7fc700 1 -- 172.21.15.78:0/3613911177 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78e4004390 con 0x7f78fc0c44c0 2023-10-14T11:58:15.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.303+0000 7f78fa7fc700 1 -- 172.21.15.78:0/3613911177 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e40036a0 con 0x7f78fc0c44c0 2023-10-14T11:58:15.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.303+0000 7f7901bee700 1 -- 172.21.15.78:0/3613911177 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78fc0c44c0 msgr2=0x7f78fc0c7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:15.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.303+0000 7f7901bee700 1 --2- 172.21.15.78:0/3613911177 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78fc0c44c0 0x7f78fc0c7a00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f78e40098f0 tx=0x7f78e4009c00 comp rx=0 tx=0).stop 2023-10-14T11:58:15.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.303+0000 7f7901bee700 1 -- 172.21.15.78:0/3613911177 shutdown_connections 2023-10-14T11:58:15.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.303+0000 7f7901bee700 1 --2- 172.21.15.78:0/3613911177 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78fc0c44c0 0x7f78fc0c7a00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:15.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.303+0000 7f7901bee700 1 -- 172.21.15.78:0/3613911177 >> 172.21.15.78:0/3613911177 conn(0x7f78fc0737d0 msgr2=0x7f78fc073be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:15.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.303+0000 7f7901bee700 1 -- 172.21.15.78:0/3613911177 shutdown_connections 2023-10-14T11:58:15.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.303+0000 7f7901bee700 1 -- 172.21.15.78:0/3613911177 wait complete. 2023-10-14T11:58:15.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.304+0000 7f7901bee700 1 Processor -- start 2023-10-14T11:58:15.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.304+0000 7f7901bee700 1 -- start start 2023-10-14T11:58:15.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.305+0000 7f7901bee700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78fc0c2c90 0x7f78fc0a8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:15.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.305+0000 7f7901bee700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f78fc0c44c0 0x7f78fc0bc710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:15.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.305+0000 7f7901bee700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f78fc0c72a0 0x7f78fc0acf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:15.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.305+0000 7f7901bee700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc06c3f0 con 0x7f78fc0c2c90 2023-10-14T11:58:15.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.305+0000 7f7901bee700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc0be800 con 0x7f78fc0c44c0 2023-10-14T11:58:15.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.305+0000 7f7901bee700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc0ab090 con 0x7f78fc0c72a0 2023-10-14T11:58:15.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.305+0000 7f78fb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78fc0c2c90 0x7f78fc0a8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:15.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.305+0000 7f78fb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78fc0c2c90 0x7f78fc0a8080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44408/0 (socket says 172.21.15.78:44408) 2023-10-14T11:58:15.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.305+0000 7f78fb7fe700 1 -- 172.21.15.78:0/1634477742 learned_addr learned my addr 172.21.15.78:0/1634477742 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:15.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.305+0000 7f78faffd700 1 --2- 172.21.15.78:0/1634477742 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f78fc0c44c0 0x7f78fc0bc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:15.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.305+0000 7f78fbfff700 1 --2- 172.21.15.78:0/1634477742 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f78fc0c72a0 0x7f78fc0acf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:15.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.305+0000 7f78fb7fe700 1 -- 172.21.15.78:0/1634477742 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f78fc0c72a0 msgr2=0x7f78fc0acf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:15.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.305+0000 7f78fb7fe700 1 --2- 172.21.15.78:0/1634477742 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f78fc0c72a0 0x7f78fc0acf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:15.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.305+0000 7f78fb7fe700 1 -- 172.21.15.78:0/1634477742 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f78fc0c44c0 msgr2=0x7f78fc0bc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:15.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.305+0000 7f78fb7fe700 1 --2- 172.21.15.78:0/1634477742 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f78fc0c44c0 0x7f78fc0bc710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:15.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.305+0000 7f78fb7fe700 1 -- 172.21.15.78:0/1634477742 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78e4009550 con 0x7f78fc0c2c90 2023-10-14T11:58:15.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.305+0000 7f78faffd700 1 --2- 172.21.15.78:0/1634477742 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f78fc0c44c0 0x7f78fc0bc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:15.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.306+0000 7f78fb7fe700 1 --2- 172.21.15.78:0/1634477742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78fc0c2c90 0x7f78fc0a8080 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f78fc0c5ce0 tx=0x7f78e4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:15.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.306+0000 7f78f3fff700 1 -- 172.21.15.78:0/1634477742 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e4003dc0 con 0x7f78fc0c2c90 2023-10-14T11:58:15.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.306+0000 7f78f3fff700 1 -- 172.21.15.78:0/1634477742 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78e401f070 con 0x7f78fc0c2c90 2023-10-14T11:58:15.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.306+0000 7f7901bee700 1 -- 172.21.15.78:0/1634477742 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78fc06c640 con 0x7f78fc0c2c90 2023-10-14T11:58:15.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.306+0000 7f78f3fff700 1 -- 172.21.15.78:0/1634477742 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e400b8a0 con 0x7f78fc0c2c90 2023-10-14T11:58:15.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.306+0000 7f7901bee700 1 -- 172.21.15.78:0/1634477742 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78fc0a85c0 con 0x7f78fc0c2c90 2023-10-14T11:58:15.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.307+0000 7f7901bee700 1 -- 172.21.15.78:0/1634477742 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78fc074b80 con 0x7f78fc0c2c90 2023-10-14T11:58:15.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.307+0000 7f78f3fff700 1 -- 172.21.15.78:0/1634477742 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f78e4021030 con 0x7f78fc0c2c90 2023-10-14T11:58:15.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.307+0000 7f78f3fff700 1 --2- 172.21.15.78:0/1634477742 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f78dc06af00 0x7f78dc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:15.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.307+0000 7f78f3fff700 1 -- 172.21.15.78:0/1634477742 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f78e408acc0 con 0x7f78fc0c2c90 2023-10-14T11:58:15.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.308+0000 7f78faffd700 1 --2- 172.21.15.78:0/1634477742 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f78dc06af00 0x7f78dc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:15.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.308+0000 7f78faffd700 1 --2- 172.21.15.78:0/1634477742 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f78dc06af00 0x7f78dc06d3c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f78fc06ce60 tx=0x7f78ec007ca0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:15.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.312+0000 7f78f3fff700 1 -- 172.21.15.78:0/1634477742 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f78e4055e00 con 0x7f78fc0c2c90 2023-10-14T11:58:15.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.505+0000 7f7901bee700 1 -- 172.21.15.78:0/1634477742 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f78fc074b80 con 0x7f78fc0c2c90 2023-10-14T11:58:15.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.507+0000 7f78f3fff700 1 -- 172.21.15.78:0/1634477742 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16494 (secure 0 0 0) 0x7f78e4059420 con 0x7f78fc0c2c90 2023-10-14T11:58:15.509 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:15.509 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":20,"fsid":"4c5dc134-3092-406a-a1e1-13e6d9e926e8","created":"2023-10-14T11:57:57.363750+0000","modified":"2023-10-14T11:58:14.534742+0000","last_up_change":"2023-10-14T11:58:06.982472+0000","last_in_change":"2023-10-14T11:57:59.821373+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T11:58:09.001200+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T11:58:12.778573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T11:58:13.490286+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"625a5fc0-9474-4eae-b80e-14c27a74ad19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6801","nonce":3884969780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6803","nonce":3884969780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6807","nonce":3884969780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6805","nonce":3884969780}]},"public_addr":"172.21.15.78:6801/3884969780","cluster_addr":"172.21.15.78:6803/3884969780","heartbeat_back_addr":"172.21.15.78:6807/3884969780","heartbeat_front_addr":"172.21.15.78:6805/3884969780","state":["exists","up"]},{"osd":1,"uuid":"49d63db1-2981-49ab-9c40-c50fc744b982","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6809","nonce":2176928406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6811","nonce":2176928406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6815","nonce":2176928406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6813","nonce":2176928406}]},"public_addr":"172.21.15.78:6809/2176928406","cluster_addr":"172.21.15.78:6811/2176928406","heartbeat_back_addr":"172.21.15.78:6815/2176928406","heartbeat_front_addr":"172.21.15.78:6813/2176928406","state":["exists","up"]},{"osd":2,"uuid":"1d7d0b29-cbd1-4717-9fe6-765a8d4998b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6825","nonce":3647044577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6827","nonce":3647044577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6831","nonce":3647044577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6829","nonce":3647044577}]},"public_addr":"172.21.15.78:6825/3647044577","cluster_addr":"172.21.15.78:6827/3647044577","heartbeat_back_addr":"172.21.15.78:6831/3647044577","heartbeat_front_addr":"172.21.15.78:6829/3647044577","state":["exists","up"]},{"osd":3,"uuid":"2f75ded1-e9e2-40e7-99ae-45e1e425e1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6817","nonce":3584112288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6819","nonce":3584112288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6823","nonce":3584112288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6821","nonce":3584112288}]},"public_addr":"172.21.15.78:6817/3584112288","cluster_addr":"172.21.15.78:6819/3584112288","heartbeat_back_addr":"172.21.15.78:6823/3584112288","heartbeat_front_addr":"172.21.15.78:6821/3584112288","state":["exists","up"]},{"osd":4,"uuid":"5130b432-8358-46cb-8a31-3064c976f6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6801","nonce":3732756301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6803","nonce":3732756301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6807","nonce":3732756301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6805","nonce":3732756301}]},"public_addr":"172.21.15.195:6801/3732756301","cluster_addr":"172.21.15.195:6803/3732756301","heartbeat_back_addr":"172.21.15.195:6807/3732756301","heartbeat_front_addr":"172.21.15.195:6805/3732756301","state":["exists","up"]},{"osd":5,"uuid":"60e234a9-a506-4e40-9c19-49a58f3cf28c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6825","nonce":2857706411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6827","nonce":2857706411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6831","nonce":2857706411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6829","nonce":2857706411}]},"public_addr":"172.21.15.195:6825/2857706411","cluster_addr":"172.21.15.195:6827/2857706411","heartbeat_back_addr":"172.21.15.195:6831/2857706411","heartbeat_front_addr":"172.21.15.195:6829/2857706411","state":["exists","up"]},{"osd":6,"uuid":"df7d2dd6-26b6-4eb1-96e5-94d1f3955b33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6810","nonce":72554374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6813","nonce":72554374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6820","nonce":72554374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6817","nonce":72554374}]},"public_addr":"172.21.15.195:6810/72554374","cluster_addr":"172.21.15.195:6813/72554374","heartbeat_back_addr":"172.21.15.195:6820/72554374","heartbeat_front_addr":"172.21.15.195:6817/72554374","state":["exists","up"]},{"osd":7,"uuid":"47a9d4de-f56d-4aa4-9900-c4a9cf9b2f97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6809","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6811","nonce":482472979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6816","nonce":482472979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6823","nonce":482472979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6821","nonce":482472979}]},"public_addr":"172.21.15.195:6811/482472979","cluster_addr":"172.21.15.195:6816/482472979","heartbeat_back_addr":"172.21.15.195:6823/482472979","heartbeat_front_addr":"172.21.15.195:6821/482472979","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.754003+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.005994+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.043603+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.024506+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.062585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.033767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.994825+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.038300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T11:58:15.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.509+0000 7f7901bee700 1 -- 172.21.15.78:0/1634477742 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f78dc06af00 msgr2=0x7f78dc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:15.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.509+0000 7f7901bee700 1 --2- 172.21.15.78:0/1634477742 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f78dc06af00 0x7f78dc06d3c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f78fc06ce60 tx=0x7f78ec007ca0 comp rx=0 tx=0).stop 2023-10-14T11:58:15.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.509+0000 7f7901bee700 1 -- 172.21.15.78:0/1634477742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78fc0c2c90 msgr2=0x7f78fc0a8080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:15.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.509+0000 7f7901bee700 1 --2- 172.21.15.78:0/1634477742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78fc0c2c90 0x7f78fc0a8080 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f78fc0c5ce0 tx=0x7f78e4004750 comp rx=0 tx=0).stop 2023-10-14T11:58:15.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.509+0000 7f7901bee700 1 -- 172.21.15.78:0/1634477742 shutdown_connections 2023-10-14T11:58:15.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.509+0000 7f7901bee700 1 --2- 172.21.15.78:0/1634477742 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f78fc0c72a0 0x7f78fc0acf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:15.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.509+0000 7f7901bee700 1 --2- 172.21.15.78:0/1634477742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78fc0c2c90 0x7f78fc0a8080 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:15.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.509+0000 7f7901bee700 1 --2- 172.21.15.78:0/1634477742 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f78dc06af00 0x7f78dc06d3c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:15.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.509+0000 7f7901bee700 1 --2- 172.21.15.78:0/1634477742 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f78fc0c44c0 0x7f78fc0bc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:15.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.509+0000 7f7901bee700 1 -- 172.21.15.78:0/1634477742 >> 172.21.15.78:0/1634477742 conn(0x7f78fc0737d0 msgr2=0x7f78fc153ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:15.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.509+0000 7f7901bee700 1 -- 172.21.15.78:0/1634477742 shutdown_connections 2023-10-14T11:58:15.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.509+0000 7f7901bee700 1 -- 172.21.15.78:0/1634477742 wait complete. 2023-10-14T11:58:15.523 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T11:58:15.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.640+0000 7f3a000ac700 1 Processor -- start 2023-10-14T11:58:15.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.640+0000 7f3a000ac700 1 -- start start 2023-10-14T11:58:15.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.641+0000 7f3a000ac700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f39f814b6b0 0x7f39f8072580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:15.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.641+0000 7f3a000ac700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f39f8154bb0 0x7f39f8155040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:15.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.641+0000 7f3a000ac700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39f80c25e0 con 0x7f39f806db50 2023-10-14T11:58:15.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.641+0000 7f3a000ac700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39f80a87b0 con 0x7f39f814b6b0 2023-10-14T11:58:15.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.641+0000 7f3a000ac700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39f814ba90 con 0x7f39f8154bb0 2023-10-14T11:58:15.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.641+0000 7f39fde48700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f39f814b6b0 0x7f39f8072580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:15.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.641+0000 7f39fde48700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f39f814b6b0 0x7f39f8072580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44416/0 (socket says 172.21.15.78:44416) 2023-10-14T11:58:15.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.641+0000 7f39fde48700 1 -- 172.21.15.78:0/3378553867 learned_addr learned my addr 172.21.15.78:0/3378553867 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:15.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.641+0000 7f39fe649700 1 --2- 172.21.15.78:0/3378553867 >> v2:172.21.15.195:3300/0 conn(0x7f39f8154bb0 0x7f39f8155040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:15.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.641+0000 7f39fde48700 1 -- 172.21.15.78:0/3378553867 >> v2:172.21.15.195:3300/0 conn(0x7f39f8154bb0 msgr2=0x7f39f8155040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:15.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.641+0000 7f39fde48700 1 --2- 172.21.15.78:0/3378553867 >> v2:172.21.15.195:3300/0 conn(0x7f39f8154bb0 0x7f39f8155040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:15.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.641+0000 7f39fde48700 1 -- 172.21.15.78:0/3378553867 >> v1:172.21.15.195:6789/0 conn(0x7f39f806db50 legacy=0x7f39f80c2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:15.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.641+0000 7f39fde48700 1 -- 172.21.15.78:0/3378553867 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f8155660 con 0x7f39f814b6b0 2023-10-14T11:58:15.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.641+0000 7f39fe649700 1 --2- 172.21.15.78:0/3378553867 >> v2:172.21.15.195:3300/0 conn(0x7f39f8154bb0 0x7f39f8155040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:15.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.641+0000 7f39fde48700 1 --2- 172.21.15.78:0/3378553867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39f814b6b0 0x7f39f8072580 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f39f814be30 tx=0x7f39f4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a65aaac87bbf4de server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:15.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.641+0000 7f39fce46700 1 -- 172.21.15.78:0/3378553867 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f4004230 con 0x7f39f814b6b0 2023-10-14T11:58:15.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.642+0000 7f39fce46700 1 -- 172.21.15.78:0/3378553867 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39f4004390 con 0x7f39f814b6b0 2023-10-14T11:58:15.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.642+0000 7f39fce46700 1 -- 172.21.15.78:0/3378553867 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f40036a0 con 0x7f39f814b6b0 2023-10-14T11:58:15.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.642+0000 7f3a000ac700 1 -- 172.21.15.78:0/3378553867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39f814b6b0 msgr2=0x7f39f8072580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:15.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.642+0000 7f3a000ac700 1 --2- 172.21.15.78:0/3378553867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39f814b6b0 0x7f39f8072580 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f39f814be30 tx=0x7f39f4009bd0 comp rx=0 tx=0).stop 2023-10-14T11:58:15.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.642+0000 7f3a000ac700 1 -- 172.21.15.78:0/3378553867 shutdown_connections 2023-10-14T11:58:15.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.642+0000 7f3a000ac700 1 --2- 172.21.15.78:0/3378553867 >> v2:172.21.15.195:3300/0 conn(0x7f39f8154bb0 0x7f39f8155040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:15.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.642+0000 7f3a000ac700 1 --2- 172.21.15.78:0/3378553867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39f814b6b0 0x7f39f8072580 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:15.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.642+0000 7f3a000ac700 1 -- 172.21.15.78:0/3378553867 >> 172.21.15.78:0/3378553867 conn(0x7f39f80c3ac0 msgr2=0x7f39f80c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:15.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.642+0000 7f3a000ac700 1 -- 172.21.15.78:0/3378553867 shutdown_connections 2023-10-14T11:58:15.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.642+0000 7f3a000ac700 1 -- 172.21.15.78:0/3378553867 wait complete. 2023-10-14T11:58:15.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.643+0000 7f3a000ac700 1 Processor -- start 2023-10-14T11:58:15.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.643+0000 7f3a000ac700 1 -- start start 2023-10-14T11:58:15.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.644+0000 7f3a000ac700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f39f806db50 0x7f39f806ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:15.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.644+0000 7f3a000ac700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39f814b6b0 0x7f39f813d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:15.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.644+0000 7f3a000ac700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f39f8154bb0 0x7f39f813d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:15.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.644+0000 7f3a000ac700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f8140850 con 0x7f39f814b6b0 2023-10-14T11:58:15.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.644+0000 7f3a000ac700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f8145db0 con 0x7f39f8154bb0 2023-10-14T11:58:15.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.644+0000 7f3a000ac700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39f8140b50 con 0x7f39f806db50 2023-10-14T11:58:15.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.644+0000 7f39fd647700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39f814b6b0 0x7f39f813d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:15.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.644+0000 7f39fd647700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39f814b6b0 0x7f39f813d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44422/0 (socket says 172.21.15.78:44422) 2023-10-14T11:58:15.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.644+0000 7f39fd647700 1 -- 172.21.15.78:0/664327902 learned_addr learned my addr 172.21.15.78:0/664327902 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:15.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.644+0000 7f39fe649700 1 --2- 172.21.15.78:0/664327902 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f39f8154bb0 0x7f39f813d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:15.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.644+0000 7f39fde48700 1 --2- 172.21.15.78:0/664327902 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f39f806db50 0x7f39f806ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:15.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.644+0000 7f39fd647700 1 -- 172.21.15.78:0/664327902 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f39f806db50 msgr2=0x7f39f806ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:15.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.644+0000 7f39fd647700 1 --2- 172.21.15.78:0/664327902 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f39f806db50 0x7f39f806ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:15.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.644+0000 7f39fd647700 1 -- 172.21.15.78:0/664327902 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f39f8154bb0 msgr2=0x7f39f813d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:15.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.644+0000 7f39fd647700 1 --2- 172.21.15.78:0/664327902 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f39f8154bb0 0x7f39f813d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:15.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.644+0000 7f39fd647700 1 -- 172.21.15.78:0/664327902 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f4009550 con 0x7f39f814b6b0 2023-10-14T11:58:15.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.644+0000 7f39fe649700 1 --2- 172.21.15.78:0/664327902 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f39f8154bb0 0x7f39f813d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:15.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.644+0000 7f39fde48700 1 --2- 172.21.15.78:0/664327902 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f39f806db50 0x7f39f806ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:15.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.645+0000 7f39fd647700 1 --2- 172.21.15.78:0/664327902 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39f814b6b0 0x7f39f813d3f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f39e800c4d0 tx=0x7f39e800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:15.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.645+0000 7f39ee7fc700 1 -- 172.21.15.78:0/664327902 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e8011790 con 0x7f39f814b6b0 2023-10-14T11:58:15.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.645+0000 7f39ee7fc700 1 -- 172.21.15.78:0/664327902 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39e80118f0 con 0x7f39f814b6b0 2023-10-14T11:58:15.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.645+0000 7f39ee7fc700 1 -- 172.21.15.78:0/664327902 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e8011ca0 con 0x7f39f814b6b0 2023-10-14T11:58:15.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.645+0000 7f3a000ac700 1 -- 172.21.15.78:0/664327902 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39f8145ad0 con 0x7f39f814b6b0 2023-10-14T11:58:15.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.645+0000 7f3a000ac700 1 -- 172.21.15.78:0/664327902 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39f806b080 con 0x7f39f814b6b0 2023-10-14T11:58:15.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.646+0000 7f3a000ac700 1 -- 172.21.15.78:0/664327902 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39f8074b80 con 0x7f39f814b6b0 2023-10-14T11:58:15.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.646+0000 7f39ee7fc700 1 -- 172.21.15.78:0/664327902 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f39e801c460 con 0x7f39f814b6b0 2023-10-14T11:58:15.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.646+0000 7f39ee7fc700 1 --2- 172.21.15.78:0/664327902 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f39dc06af00 0x7f39dc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:15.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.647+0000 7f39fde48700 1 --2- 172.21.15.78:0/664327902 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f39dc06af00 0x7f39dc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:15.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.647+0000 7f39ee7fc700 1 -- 172.21.15.78:0/664327902 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f39e80892f0 con 0x7f39f814b6b0 2023-10-14T11:58:15.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.647+0000 7f39fde48700 1 --2- 172.21.15.78:0/664327902 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f39dc06af00 0x7f39dc06d3c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f39f4009130 tx=0x7f39f4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:15.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.651+0000 7f39ee7fc700 1 -- 172.21.15.78:0/664327902 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f39e8054430 con 0x7f39f814b6b0 2023-10-14T11:58:15.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.842+0000 7f3a000ac700 1 -- 172.21.15.78:0/664327902 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39f8072360 con 0x7f39f814b6b0 2023-10-14T11:58:15.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.843+0000 7f39ee7fc700 1 -- 172.21.15.78:0/664327902 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16494 (secure 0 0 0) 0x7f39e8057a50 con 0x7f39f814b6b0 2023-10-14T11:58:15.846 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:15.846 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":20,"fsid":"4c5dc134-3092-406a-a1e1-13e6d9e926e8","created":"2023-10-14T11:57:57.363750+0000","modified":"2023-10-14T11:58:14.534742+0000","last_up_change":"2023-10-14T11:58:06.982472+0000","last_in_change":"2023-10-14T11:57:59.821373+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T11:58:09.001200+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T11:58:12.778573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T11:58:13.490286+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"625a5fc0-9474-4eae-b80e-14c27a74ad19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6801","nonce":3884969780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6803","nonce":3884969780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6807","nonce":3884969780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6805","nonce":3884969780}]},"public_addr":"172.21.15.78:6801/3884969780","cluster_addr":"172.21.15.78:6803/3884969780","heartbeat_back_addr":"172.21.15.78:6807/3884969780","heartbeat_front_addr":"172.21.15.78:6805/3884969780","state":["exists","up"]},{"osd":1,"uuid":"49d63db1-2981-49ab-9c40-c50fc744b982","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6809","nonce":2176928406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6811","nonce":2176928406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6815","nonce":2176928406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6813","nonce":2176928406}]},"public_addr":"172.21.15.78:6809/2176928406","cluster_addr":"172.21.15.78:6811/2176928406","heartbeat_back_addr":"172.21.15.78:6815/2176928406","heartbeat_front_addr":"172.21.15.78:6813/2176928406","state":["exists","up"]},{"osd":2,"uuid":"1d7d0b29-cbd1-4717-9fe6-765a8d4998b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6825","nonce":3647044577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6827","nonce":3647044577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6831","nonce":3647044577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6829","nonce":3647044577}]},"public_addr":"172.21.15.78:6825/3647044577","cluster_addr":"172.21.15.78:6827/3647044577","heartbeat_back_addr":"172.21.15.78:6831/3647044577","heartbeat_front_addr":"172.21.15.78:6829/3647044577","state":["exists","up"]},{"osd":3,"uuid":"2f75ded1-e9e2-40e7-99ae-45e1e425e1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6817","nonce":3584112288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6819","nonce":3584112288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6823","nonce":3584112288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6821","nonce":3584112288}]},"public_addr":"172.21.15.78:6817/3584112288","cluster_addr":"172.21.15.78:6819/3584112288","heartbeat_back_addr":"172.21.15.78:6823/3584112288","heartbeat_front_addr":"172.21.15.78:6821/3584112288","state":["exists","up"]},{"osd":4,"uuid":"5130b432-8358-46cb-8a31-3064c976f6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6801","nonce":3732756301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6803","nonce":3732756301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6807","nonce":3732756301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6805","nonce":3732756301}]},"public_addr":"172.21.15.195:6801/3732756301","cluster_addr":"172.21.15.195:6803/3732756301","heartbeat_back_addr":"172.21.15.195:6807/3732756301","heartbeat_front_addr":"172.21.15.195:6805/3732756301","state":["exists","up"]},{"osd":5,"uuid":"60e234a9-a506-4e40-9c19-49a58f3cf28c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6825","nonce":2857706411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6827","nonce":2857706411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6831","nonce":2857706411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6829","nonce":2857706411}]},"public_addr":"172.21.15.195:6825/2857706411","cluster_addr":"172.21.15.195:6827/2857706411","heartbeat_back_addr":"172.21.15.195:6831/2857706411","heartbeat_front_addr":"172.21.15.195:6829/2857706411","state":["exists","up"]},{"osd":6,"uuid":"df7d2dd6-26b6-4eb1-96e5-94d1f3955b33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6810","nonce":72554374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6813","nonce":72554374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6820","nonce":72554374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6817","nonce":72554374}]},"public_addr":"172.21.15.195:6810/72554374","cluster_addr":"172.21.15.195:6813/72554374","heartbeat_back_addr":"172.21.15.195:6820/72554374","heartbeat_front_addr":"172.21.15.195:6817/72554374","state":["exists","up"]},{"osd":7,"uuid":"47a9d4de-f56d-4aa4-9900-c4a9cf9b2f97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6809","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6811","nonce":482472979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6816","nonce":482472979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6823","nonce":482472979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6821","nonce":482472979}]},"public_addr":"172.21.15.195:6811/482472979","cluster_addr":"172.21.15.195:6816/482472979","heartbeat_back_addr":"172.21.15.195:6823/482472979","heartbeat_front_addr":"172.21.15.195:6821/482472979","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.754003+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.005994+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.043603+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.024506+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.062585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.033767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.994825+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.038300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T11:58:15.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.845+0000 7f3a000ac700 1 -- 172.21.15.78:0/664327902 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f39dc06af00 msgr2=0x7f39dc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:15.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.845+0000 7f3a000ac700 1 --2- 172.21.15.78:0/664327902 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f39dc06af00 0x7f39dc06d3c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f39f4009130 tx=0x7f39f4008030 comp rx=0 tx=0).stop 2023-10-14T11:58:15.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.845+0000 7f3a000ac700 1 -- 172.21.15.78:0/664327902 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39f814b6b0 msgr2=0x7f39f813d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:15.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.845+0000 7f3a000ac700 1 --2- 172.21.15.78:0/664327902 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39f814b6b0 0x7f39f813d3f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f39e800c4d0 tx=0x7f39e800c890 comp rx=0 tx=0).stop 2023-10-14T11:58:15.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.846+0000 7f3a000ac700 1 -- 172.21.15.78:0/664327902 shutdown_connections 2023-10-14T11:58:15.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.846+0000 7f3a000ac700 1 --2- 172.21.15.78:0/664327902 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f39dc06af00 0x7f39dc06d3c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:15.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.846+0000 7f3a000ac700 1 --2- 172.21.15.78:0/664327902 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f39f8154bb0 0x7f39f813d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:15.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.846+0000 7f3a000ac700 1 --2- 172.21.15.78:0/664327902 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f39f806db50 0x7f39f806ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:15.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.846+0000 7f3a000ac700 1 --2- 172.21.15.78:0/664327902 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39f814b6b0 0x7f39f813d3f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:15.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.846+0000 7f3a000ac700 1 -- 172.21.15.78:0/664327902 >> 172.21.15.78:0/664327902 conn(0x7f39f80c3ac0 msgr2=0x7f39f80c2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:15.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.846+0000 7f3a000ac700 1 -- 172.21.15.78:0/664327902 shutdown_connections 2023-10-14T11:58:15.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.846+0000 7f3a000ac700 1 -- 172.21.15.78:0/664327902 wait complete. 2023-10-14T11:58:15.860 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-14T11:58:15.982 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.980+0000 7fe8c33ae700 1 Processor -- start 2023-10-14T11:58:15.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.980+0000 7fe8c33ae700 1 -- start start 2023-10-14T11:58:15.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.981+0000 7fe8c33ae700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe8bc13dbe0 0x7fe8bc06a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:15.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.981+0000 7fe8c33ae700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe8bc0c4540 0x7fe8bc0c02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:15.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.981+0000 7fe8c33ae700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe8bc06ae10 0x7fe8bc0c0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:15.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.981+0000 7fe8c33ae700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8bc06d330 con 0x7fe8bc06ae10 2023-10-14T11:58:15.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.981+0000 7fe8c33ae700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8bc06ebe0 con 0x7fe8bc13dbe0 2023-10-14T11:58:15.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.981+0000 7fe8c33ae700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8bc0c0d60 con 0x7fe8bc0c4540 2023-10-14T11:58:15.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.981+0000 7fe8c194b700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe8bc06ae10 0x7fe8bc0c0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:15.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.981+0000 7fe8c194b700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe8bc06ae10 0x7fe8bc0c0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44432/0 (socket says 172.21.15.78:44432) 2023-10-14T11:58:15.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.981+0000 7fe8c194b700 1 -- 172.21.15.78:0/1092206710 learned_addr learned my addr 172.21.15.78:0/1092206710 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:15.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.982+0000 7fe8c114a700 1 --2- 172.21.15.78:0/1092206710 >> v2:172.21.15.195:3300/0 conn(0x7fe8bc13dbe0 0x7fe8bc06a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:15.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.982+0000 7fe8c0949700 1 --2- 172.21.15.78:0/1092206710 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe8bc0c4540 0x7fe8bc0c02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:15.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.982+0000 7fe8c194b700 1 -- 172.21.15.78:0/1092206710 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe8bc0c4540 msgr2=0x7fe8bc0c02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:15.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.982+0000 7fe8c194b700 1 --2- 172.21.15.78:0/1092206710 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe8bc0c4540 0x7fe8bc0c02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:15.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.982+0000 7fe8c194b700 1 -- 172.21.15.78:0/1092206710 >> v2:172.21.15.195:3300/0 conn(0x7fe8bc13dbe0 msgr2=0x7fe8bc06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:15.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.982+0000 7fe8c194b700 1 --2- 172.21.15.78:0/1092206710 >> v2:172.21.15.195:3300/0 conn(0x7fe8bc13dbe0 0x7fe8bc06a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:15.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.982+0000 7fe8c194b700 1 -- 172.21.15.78:0/1092206710 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8bc1558c0 con 0x7fe8bc06ae10 2023-10-14T11:58:15.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.982+0000 7fe8c194b700 1 --2- 172.21.15.78:0/1092206710 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8bc06ae10 0x7fe8bc0c0820 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc072b60 tx=0x7fe8b400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4d0429fd64de8e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:15.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.982+0000 7fe8b3fff700 1 -- 172.21.15.78:0/1092206710 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b400e070 con 0x7fe8bc06ae10 2023-10-14T11:58:15.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.982+0000 7fe8b3fff700 1 -- 172.21.15.78:0/1092206710 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8b400b9c0 con 0x7fe8bc06ae10 2023-10-14T11:58:15.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.982+0000 7fe8b3fff700 1 -- 172.21.15.78:0/1092206710 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b400bd70 con 0x7fe8bc06ae10 2023-10-14T11:58:15.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.982+0000 7fe8c33ae700 1 -- 172.21.15.78:0/1092206710 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8bc06ae10 msgr2=0x7fe8bc0c0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:15.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.982+0000 7fe8c33ae700 1 --2- 172.21.15.78:0/1092206710 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8bc06ae10 0x7fe8bc0c0820 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc072b60 tx=0x7fe8b400b330 comp rx=0 tx=0).stop 2023-10-14T11:58:15.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.982+0000 7fe8c33ae700 1 -- 172.21.15.78:0/1092206710 shutdown_connections 2023-10-14T11:58:15.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.982+0000 7fe8c33ae700 1 --2- 172.21.15.78:0/1092206710 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8bc06ae10 0x7fe8bc0c0820 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:15.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.982+0000 7fe8c33ae700 1 --2- 172.21.15.78:0/1092206710 >> v2:172.21.15.195:3300/0 conn(0x7fe8bc13dbe0 0x7fe8bc06a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:15.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.982+0000 7fe8c33ae700 1 --2- 172.21.15.78:0/1092206710 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe8bc0c4540 0x7fe8bc0c02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:15.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.982+0000 7fe8c33ae700 1 -- 172.21.15.78:0/1092206710 >> 172.21.15.78:0/1092206710 conn(0x7fe8bc0c3a50 msgr2=0x7fe8bc0c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:15.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.983+0000 7fe8c33ae700 1 -- 172.21.15.78:0/1092206710 shutdown_connections 2023-10-14T11:58:15.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.983+0000 7fe8c33ae700 1 -- 172.21.15.78:0/1092206710 wait complete. 2023-10-14T11:58:15.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.984+0000 7fe8c33ae700 1 Processor -- start 2023-10-14T11:58:15.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.984+0000 7fe8c33ae700 1 -- start start 2023-10-14T11:58:15.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.984+0000 7fe8c33ae700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8bc06ae10 0x7fe8bc155df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:15.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.984+0000 7fe8c33ae700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe8bc0c4540 0x7fe8bc0c6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:15.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.984+0000 7fe8c33ae700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe8bc13dbe0 0x7fe8bc0c27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:15.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.984+0000 7fe8c33ae700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8bc157ad0 con 0x7fe8bc06ae10 2023-10-14T11:58:15.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.984+0000 7fe8c33ae700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8bc0c2320 con 0x7fe8bc0c4540 2023-10-14T11:58:15.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.984+0000 7fe8c33ae700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8bc0c19f0 con 0x7fe8bc13dbe0 2023-10-14T11:58:15.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.984+0000 7fe8c114a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8bc06ae10 0x7fe8bc155df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:15.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.985+0000 7fe8c114a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8bc06ae10 0x7fe8bc155df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44436/0 (socket says 172.21.15.78:44436) 2023-10-14T11:58:15.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.985+0000 7fe8c114a700 1 -- 172.21.15.78:0/869188246 learned_addr learned my addr 172.21.15.78:0/869188246 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:15.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.985+0000 7fe8c0949700 1 --2- 172.21.15.78:0/869188246 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe8bc0c4540 0x7fe8bc0c6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:15.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.985+0000 7fe8c194b700 1 --2- 172.21.15.78:0/869188246 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe8bc13dbe0 0x7fe8bc0c27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:15.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.985+0000 7fe8c114a700 1 -- 172.21.15.78:0/869188246 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe8bc13dbe0 msgr2=0x7fe8bc0c27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:15.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.985+0000 7fe8c114a700 1 --2- 172.21.15.78:0/869188246 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe8bc13dbe0 0x7fe8bc0c27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:15.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.985+0000 7fe8c114a700 1 -- 172.21.15.78:0/869188246 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe8bc0c4540 msgr2=0x7fe8bc0c6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:15.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.985+0000 7fe8c114a700 1 --2- 172.21.15.78:0/869188246 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe8bc0c4540 0x7fe8bc0c6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:15.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.985+0000 7fe8c114a700 1 -- 172.21.15.78:0/869188246 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b4009b30 con 0x7fe8bc06ae10 2023-10-14T11:58:15.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.985+0000 7fe8c194b700 1 --2- 172.21.15.78:0/869188246 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe8bc13dbe0 0x7fe8bc0c27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:15.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.985+0000 7fe8c0949700 1 --2- 172.21.15.78:0/869188246 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe8bc0c4540 0x7fe8bc0c6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:15.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.985+0000 7fe8c114a700 1 --2- 172.21.15.78:0/869188246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8bc06ae10 0x7fe8bc155df0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac00c820 tx=0x7fe8ac00cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:15.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.985+0000 7fe8b1ffb700 1 -- 172.21.15.78:0/869188246 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac011800 con 0x7fe8bc06ae10 2023-10-14T11:58:15.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.985+0000 7fe8b1ffb700 1 -- 172.21.15.78:0/869188246 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8ac011960 con 0x7fe8bc06ae10 2023-10-14T11:58:15.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.985+0000 7fe8c33ae700 1 -- 172.21.15.78:0/869188246 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8bc0c1cd0 con 0x7fe8bc06ae10 2023-10-14T11:58:15.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.985+0000 7fe8b1ffb700 1 -- 172.21.15.78:0/869188246 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac011d10 con 0x7fe8bc06ae10 2023-10-14T11:58:15.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.985+0000 7fe8c33ae700 1 -- 172.21.15.78:0/869188246 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8bc0c65d0 con 0x7fe8bc06ae10 2023-10-14T11:58:15.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.986+0000 7fe8c33ae700 1 -- 172.21.15.78:0/869188246 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8bc06de20 con 0x7fe8bc06ae10 2023-10-14T11:58:15.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.987+0000 7fe8b1ffb700 1 -- 172.21.15.78:0/869188246 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe8ac024480 con 0x7fe8bc06ae10 2023-10-14T11:58:16.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.988+0000 7fe8b1ffb700 1 --2- 172.21.15.78:0/869188246 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe8a006af00 0x7fe8a006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:16.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.988+0000 7fe8b1ffb700 1 -- 172.21.15.78:0/869188246 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe8ac0899a0 con 0x7fe8bc06ae10 2023-10-14T11:58:16.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.988+0000 7fe8c0949700 1 --2- 172.21.15.78:0/869188246 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe8a006af00 0x7fe8a006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:16.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.988+0000 7fe8c0949700 1 --2- 172.21.15.78:0/869188246 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe8a006af00 0x7fe8a006d3c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc1534a0 tx=0x7fe8b8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:16.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:15.991+0000 7fe8b1ffb700 1 -- 172.21.15.78:0/869188246 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe8ac054ac0 con 0x7fe8bc06ae10 2023-10-14T11:58:16.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:16.229+0000 7fe8c33ae700 1 -- 172.21.15.78:0/869188246 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe8bc074b80 con 0x7fe8bc06ae10 2023-10-14T11:58:17.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.166+0000 7fe8b1ffb700 1 -- 172.21.15.78:0/869188246 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe8ac0580e0 con 0x7fe8bc06ae10 2023-10-14T11:58:17.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.169+0000 7fe8c33ae700 1 -- 172.21.15.78:0/869188246 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe8a006af00 msgr2=0x7fe8a006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:17.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.169+0000 7fe8c33ae700 1 --2- 172.21.15.78:0/869188246 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe8a006af00 0x7fe8a006d3c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc1534a0 tx=0x7fe8b8008040 comp rx=0 tx=0).stop 2023-10-14T11:58:17.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.169+0000 7fe8c33ae700 1 -- 172.21.15.78:0/869188246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8bc06ae10 msgr2=0x7fe8bc155df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:17.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.169+0000 7fe8c33ae700 1 --2- 172.21.15.78:0/869188246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8bc06ae10 0x7fe8bc155df0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac00c820 tx=0x7fe8ac00cbe0 comp rx=0 tx=0).stop 2023-10-14T11:58:17.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.169+0000 7fe8c33ae700 1 -- 172.21.15.78:0/869188246 shutdown_connections 2023-10-14T11:58:17.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.169+0000 7fe8c33ae700 1 --2- 172.21.15.78:0/869188246 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe8bc13dbe0 0x7fe8bc0c27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:17.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.169+0000 7fe8c33ae700 1 --2- 172.21.15.78:0/869188246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8bc06ae10 0x7fe8bc155df0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:17.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.169+0000 7fe8c33ae700 1 --2- 172.21.15.78:0/869188246 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe8a006af00 0x7fe8a006d3c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:17.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.169+0000 7fe8c33ae700 1 --2- 172.21.15.78:0/869188246 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe8bc0c4540 0x7fe8bc0c6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:17.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.169+0000 7fe8c33ae700 1 -- 172.21.15.78:0/869188246 >> 172.21.15.78:0/869188246 conn(0x7fe8bc0c3a50 msgr2=0x7fe8bc154280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:17.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.169+0000 7fe8c33ae700 1 -- 172.21.15.78:0/869188246 shutdown_connections 2023-10-14T11:58:17.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.170+0000 7fe8c33ae700 1 -- 172.21.15.78:0/869188246 wait complete. 2023-10-14T11:58:17.186 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-14T11:58:17.186 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-14T11:58:17.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.312+0000 7fbc90b0f700 1 Processor -- start 2023-10-14T11:58:17.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.312+0000 7fbc90b0f700 1 -- start start 2023-10-14T11:58:17.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.313+0000 7fbc90b0f700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fbc8c0c69e0 0x7fbc8c06e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:17.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.313+0000 7fbc90b0f700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc8c1542e0 con 0x7fbc8c06e840 2023-10-14T11:58:17.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.313+0000 7fbc90b0f700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc8c1519f0 con 0x7fbc8c0c11a0 2023-10-14T11:58:17.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.313+0000 7fbc90b0f700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc8c151f20 con 0x7fbc8c0c69e0 2023-10-14T11:58:17.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.313+0000 7fbc8a59c700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fbc8c0c69e0 0x7fbc8c06e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:17.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.313+0000 7fbc89d9b700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fbc8c06e840 0x7fbc8c14dfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49320/0 (socket says 172.21.15.78:49320) 2023-10-14T11:58:17.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.313+0000 7fbc89d9b700 1 -- 172.21.15.78:0/2408713875 learned_addr learned my addr 172.21.15.78:0/2408713875 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:17.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.313+0000 7fbc8a59c700 1 -- 172.21.15.78:0/2408713875 >> v1:172.21.15.195:6789/0 conn(0x7fbc8c0c11a0 legacy=0x7fbc8c0ab960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:17.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.313+0000 7fbc8a59c700 1 -- 172.21.15.78:0/2408713875 >> v1:172.21.15.78:6789/0 conn(0x7fbc8c06e840 legacy=0x7fbc8c14dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:17.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.313+0000 7fbc8959a700 1 -- 172.21.15.78:0/2408713875 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2339942775 0 0) 0x7fbc8c1542e0 con 0x7fbc8c06e840 2023-10-14T11:58:17.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.313+0000 7fbc8a59c700 1 -- 172.21.15.78:0/2408713875 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc8c070c50 con 0x7fbc8c0c69e0 2023-10-14T11:58:17.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.313+0000 7fbc8a59c700 1 --2- 172.21.15.78:0/2408713875 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc8c0c69e0 0x7fbc8c06e300 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c009840 tx=0x7fbc7c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91111153aacf1cfb server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:17.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.313+0000 7fbc8959a700 1 -- 172.21.15.78:0/2408713875 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7c011830 con 0x7fbc8c0c69e0 2023-10-14T11:58:17.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.313+0000 7fbc8959a700 1 -- 172.21.15.78:0/2408713875 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc7c011990 con 0x7fbc8c0c69e0 2023-10-14T11:58:17.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.313+0000 7fbc8959a700 1 -- 172.21.15.78:0/2408713875 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7c011d20 con 0x7fbc8c0c69e0 2023-10-14T11:58:17.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.314+0000 7fbc90b0f700 1 -- 172.21.15.78:0/2408713875 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc8c0c69e0 msgr2=0x7fbc8c06e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:17.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.314+0000 7fbc90b0f700 1 --2- 172.21.15.78:0/2408713875 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc8c0c69e0 0x7fbc8c06e300 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c009840 tx=0x7fbc7c009b50 comp rx=0 tx=0).stop 2023-10-14T11:58:17.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.314+0000 7fbc90b0f700 1 -- 172.21.15.78:0/2408713875 shutdown_connections 2023-10-14T11:58:17.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.314+0000 7fbc90b0f700 1 --2- 172.21.15.78:0/2408713875 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc8c0c69e0 0x7fbc8c06e300 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:17.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.314+0000 7fbc90b0f700 1 -- 172.21.15.78:0/2408713875 >> 172.21.15.78:0/2408713875 conn(0x7fbc8c0a8410 msgr2=0x7fbc8c0a8820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:17.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.314+0000 7fbc90b0f700 1 -- 172.21.15.78:0/2408713875 shutdown_connections 2023-10-14T11:58:17.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.314+0000 7fbc90b0f700 1 -- 172.21.15.78:0/2408713875 wait complete. 2023-10-14T11:58:17.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.315+0000 7fbc90b0f700 1 Processor -- start 2023-10-14T11:58:17.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.315+0000 7fbc90b0f700 1 -- start start 2023-10-14T11:58:17.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.316+0000 7fbc90b0f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc8c06e840 0x7fbc8c0b9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:17.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.316+0000 7fbc90b0f700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbc8c0c11a0 0x7fbc8c0be420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:17.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.316+0000 7fbc90b0f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc8c0c69e0 0x7fbc8c0be960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:17.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.316+0000 7fbc90b0f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c0bd280 con 0x7fbc8c0c69e0 2023-10-14T11:58:17.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.316+0000 7fbc90b0f700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c0bd560 con 0x7fbc8c06e840 2023-10-14T11:58:17.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.316+0000 7fbc90b0f700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c0c0330 con 0x7fbc8c0c11a0 2023-10-14T11:58:17.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.316+0000 7fbc8ad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc8c0c69e0 0x7fbc8c0be960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:17.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.316+0000 7fbc8ad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc8c0c69e0 0x7fbc8c0be960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44452/0 (socket says 172.21.15.78:44452) 2023-10-14T11:58:17.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.316+0000 7fbc8ad9d700 1 -- 172.21.15.78:0/2852300723 learned_addr learned my addr 172.21.15.78:0/2852300723 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:17.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.316+0000 7fbc8a59c700 1 --2- 172.21.15.78:0/2852300723 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc8c06e840 0x7fbc8c0b9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:17.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.316+0000 7fbc89d9b700 1 --2- 172.21.15.78:0/2852300723 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbc8c0c11a0 0x7fbc8c0be420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:17.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.316+0000 7fbc8ad9d700 1 -- 172.21.15.78:0/2852300723 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbc8c0c11a0 msgr2=0x7fbc8c0be420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:17.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.316+0000 7fbc8ad9d700 1 --2- 172.21.15.78:0/2852300723 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbc8c0c11a0 0x7fbc8c0be420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:17.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.317+0000 7fbc8ad9d700 1 -- 172.21.15.78:0/2852300723 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc8c06e840 msgr2=0x7fbc8c0b9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:17.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.317+0000 7fbc8ad9d700 1 --2- 172.21.15.78:0/2852300723 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc8c06e840 0x7fbc8c0b9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:17.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.317+0000 7fbc8ad9d700 1 -- 172.21.15.78:0/2852300723 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc7c0094a0 con 0x7fbc8c0c69e0 2023-10-14T11:58:17.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.317+0000 7fbc8a59c700 1 --2- 172.21.15.78:0/2852300723 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc8c06e840 0x7fbc8c0b9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:17.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.317+0000 7fbc89d9b700 1 --2- 172.21.15.78:0/2852300723 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbc8c0c11a0 0x7fbc8c0be420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:17.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.317+0000 7fbc8ad9d700 1 --2- 172.21.15.78:0/2852300723 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc8c0c69e0 0x7fbc8c0be960 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbc8400cc50 tx=0x7fbc8400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:17.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.317+0000 7fbc7affd700 1 -- 172.21.15.78:0/2852300723 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc8400ad80 con 0x7fbc8c0c69e0 2023-10-14T11:58:17.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.317+0000 7fbc7affd700 1 -- 172.21.15.78:0/2852300723 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc8400ce50 con 0x7fbc8c0c69e0 2023-10-14T11:58:17.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.317+0000 7fbc90b0f700 1 -- 172.21.15.78:0/2852300723 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc8c0c0610 con 0x7fbc8c0c69e0 2023-10-14T11:58:17.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.317+0000 7fbc7affd700 1 -- 172.21.15.78:0/2852300723 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc84003c40 con 0x7fbc8c0c69e0 2023-10-14T11:58:17.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.317+0000 7fbc90b0f700 1 -- 172.21.15.78:0/2852300723 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc8c14fff0 con 0x7fbc8c0c69e0 2023-10-14T11:58:17.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.318+0000 7fbc90b0f700 1 -- 172.21.15.78:0/2852300723 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc8c0bdc30 con 0x7fbc8c0c69e0 2023-10-14T11:58:17.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.318+0000 7fbc7affd700 1 -- 172.21.15.78:0/2852300723 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fbc840050d0 con 0x7fbc8c0c69e0 2023-10-14T11:58:17.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.319+0000 7fbc7affd700 1 --2- 172.21.15.78:0/2852300723 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fbc6c06af00 0x7fbc6c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:17.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.319+0000 7fbc7affd700 1 -- 172.21.15.78:0/2852300723 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbc84089660 con 0x7fbc8c0c69e0 2023-10-14T11:58:17.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.319+0000 7fbc8a59c700 1 --2- 172.21.15.78:0/2852300723 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fbc6c06af00 0x7fbc6c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:17.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.319+0000 7fbc8a59c700 1 --2- 172.21.15.78:0/2852300723 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fbc6c06af00 0x7fbc6c06d3c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c000c00 tx=0x7fbc7c01c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:17.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.323+0000 7fbc7affd700 1 -- 172.21.15.78:0/2852300723 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbc8c0bdc30 con 0x7fbc8c0c69e0 2023-10-14T11:58:17.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:17.557+0000 7fbc90b0f700 1 -- 172.21.15.78:0/2852300723 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fbc8c0bdc30 con 0x7fbc8c0c69e0 2023-10-14T11:58:18.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.167+0000 7fbc7affd700 1 -- 172.21.15.78:0/2852300723 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7fbc8c0bdc30 con 0x7fbc8c0c69e0 2023-10-14T11:58:18.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.170+0000 7fbc90b0f700 1 -- 172.21.15.78:0/2852300723 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fbc6c06af00 msgr2=0x7fbc6c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:18.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.170+0000 7fbc90b0f700 1 --2- 172.21.15.78:0/2852300723 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fbc6c06af00 0x7fbc6c06d3c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c000c00 tx=0x7fbc7c01c040 comp rx=0 tx=0).stop 2023-10-14T11:58:18.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.170+0000 7fbc90b0f700 1 -- 172.21.15.78:0/2852300723 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc8c0c69e0 msgr2=0x7fbc8c0be960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:18.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.170+0000 7fbc90b0f700 1 --2- 172.21.15.78:0/2852300723 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc8c0c69e0 0x7fbc8c0be960 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbc8400cc50 tx=0x7fbc8400a340 comp rx=0 tx=0).stop 2023-10-14T11:58:18.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.170+0000 7fbc90b0f700 1 -- 172.21.15.78:0/2852300723 shutdown_connections 2023-10-14T11:58:18.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.170+0000 7fbc90b0f700 1 --2- 172.21.15.78:0/2852300723 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc8c0c69e0 0x7fbc8c0be960 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:18.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.170+0000 7fbc90b0f700 1 --2- 172.21.15.78:0/2852300723 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fbc6c06af00 0x7fbc6c06d3c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:18.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.170+0000 7fbc90b0f700 1 --2- 172.21.15.78:0/2852300723 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc8c06e840 0x7fbc8c0b9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:18.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.170+0000 7fbc90b0f700 1 --2- 172.21.15.78:0/2852300723 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbc8c0c11a0 0x7fbc8c0be420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:18.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.170+0000 7fbc90b0f700 1 -- 172.21.15.78:0/2852300723 >> 172.21.15.78:0/2852300723 conn(0x7fbc8c0a8410 msgr2=0x7fbc8c0c4620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:18.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.170+0000 7fbc90b0f700 1 -- 172.21.15.78:0/2852300723 shutdown_connections 2023-10-14T11:58:18.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.170+0000 7fbc90b0f700 1 -- 172.21.15.78:0/2852300723 wait complete. 2023-10-14T11:58:18.186 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-14T11:58:18.186 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T11:58:18.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.308+0000 7f69e5647700 1 Processor -- start 2023-10-14T11:58:18.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.308+0000 7f69e5647700 1 -- start start 2023-10-14T11:58:18.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.309+0000 7f69e5647700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f69e0154b90 0x7f69e00c5150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:18.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.309+0000 7f69e5647700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f69e006cd00 0x7f69e006b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:18.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.309+0000 7f69e5647700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69e00c9970 con 0x7f69e014b7e0 2023-10-14T11:58:18.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.309+0000 7f69e5647700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69e00c4e40 con 0x7f69e006cd00 2023-10-14T11:58:18.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.309+0000 7f69e5647700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69e00c81c0 con 0x7f69e0154b90 2023-10-14T11:58:18.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.309+0000 7f69deffd700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f69e014b7e0 0x7f69e00c9260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49332/0 (socket says 172.21.15.78:49332) 2023-10-14T11:58:18.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.309+0000 7f69df7fe700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f69e006cd00 0x7f69e006b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:18.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.309+0000 7f69deffd700 1 -- 172.21.15.78:0/3954465342 learned_addr learned my addr 172.21.15.78:0/3954465342 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:18.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.309+0000 7f69de7fc700 1 --2- 172.21.15.78:0/3954465342 >> v2:172.21.15.195:3300/0 conn(0x7f69e0154b90 0x7f69e00c5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:18.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.310+0000 7f69ddffb700 1 -- 172.21.15.78:0/3954465342 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 930607511 0 0) 0x7f69e00c9970 con 0x7f69e014b7e0 2023-10-14T11:58:18.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.310+0000 7f69ddffb700 1 -- 172.21.15.78:0/3954465342 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69c80035b0 con 0x7f69e014b7e0 2023-10-14T11:58:18.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.310+0000 7f69df7fe700 1 -- 172.21.15.78:0/3954465342 >> v2:172.21.15.195:3300/0 conn(0x7f69e0154b90 msgr2=0x7f69e00c5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:18.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.310+0000 7f69df7fe700 1 --2- 172.21.15.78:0/3954465342 >> v2:172.21.15.195:3300/0 conn(0x7f69e0154b90 0x7f69e00c5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:18.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.310+0000 7f69df7fe700 1 -- 172.21.15.78:0/3954465342 >> v1:172.21.15.78:6789/0 conn(0x7f69e014b7e0 legacy=0x7f69e00c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:18.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.310+0000 7f69df7fe700 1 -- 172.21.15.78:0/3954465342 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e00c84a0 con 0x7f69e006cd00 2023-10-14T11:58:18.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.310+0000 7f69df7fe700 1 --2- 172.21.15.78:0/3954465342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69e006cd00 0x7f69e006b440 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f69e00c1160 tx=0x7f69d800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18eee651f4eae4ee server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:18.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.310+0000 7f69ddffb700 1 -- 172.21.15.78:0/3954465342 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d800e070 con 0x7f69e006cd00 2023-10-14T11:58:18.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.310+0000 7f69ddffb700 1 -- 172.21.15.78:0/3954465342 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d800bb20 con 0x7f69e006cd00 2023-10-14T11:58:18.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.310+0000 7f69ddffb700 1 -- 172.21.15.78:0/3954465342 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d8012430 con 0x7f69e006cd00 2023-10-14T11:58:18.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.311+0000 7f69e5647700 1 -- 172.21.15.78:0/3954465342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69e006cd00 msgr2=0x7f69e006b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:18.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.311+0000 7f69e5647700 1 --2- 172.21.15.78:0/3954465342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69e006cd00 0x7f69e006b440 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f69e00c1160 tx=0x7f69d800b330 comp rx=0 tx=0).stop 2023-10-14T11:58:18.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.311+0000 7f69e5647700 1 -- 172.21.15.78:0/3954465342 shutdown_connections 2023-10-14T11:58:18.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.311+0000 7f69e5647700 1 --2- 172.21.15.78:0/3954465342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69e006cd00 0x7f69e006b440 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:18.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.311+0000 7f69e5647700 1 --2- 172.21.15.78:0/3954465342 >> v2:172.21.15.195:3300/0 conn(0x7f69e0154b90 0x7f69e00c5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:18.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.311+0000 7f69e5647700 1 -- 172.21.15.78:0/3954465342 >> 172.21.15.78:0/3954465342 conn(0x7f69e00ca2c0 msgr2=0x7f69e00ca6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:18.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.311+0000 7f69e5647700 1 -- 172.21.15.78:0/3954465342 shutdown_connections 2023-10-14T11:58:18.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.311+0000 7f69e5647700 1 -- 172.21.15.78:0/3954465342 wait complete. 2023-10-14T11:58:18.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.312+0000 7f69e5647700 1 Processor -- start 2023-10-14T11:58:18.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.312+0000 7f69e5647700 1 -- start start 2023-10-14T11:58:18.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.313+0000 7f69e5647700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f69e006cd00 0x7f69e0147a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:18.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.313+0000 7f69e5647700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f69e014b7e0 0x7f69e0145620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:18.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.313+0000 7f69e5647700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69e0154b90 0x7f69e0145b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:18.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.313+0000 7f69e5647700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e00c1bd0 con 0x7f69e0154b90 2023-10-14T11:58:18.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.313+0000 7f69e5647700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e01583e0 con 0x7f69e014b7e0 2023-10-14T11:58:18.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.313+0000 7f69e5647700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e0157340 con 0x7f69e006cd00 2023-10-14T11:58:18.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.313+0000 7f69df7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69e0154b90 0x7f69e0145b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:18.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.313+0000 7f69df7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69e0154b90 0x7f69e0145b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44466/0 (socket says 172.21.15.78:44466) 2023-10-14T11:58:18.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.313+0000 7f69df7fe700 1 -- 172.21.15.78:0/1683065398 learned_addr learned my addr 172.21.15.78:0/1683065398 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:18.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.313+0000 7f69de7fc700 1 --2- 172.21.15.78:0/1683065398 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f69e014b7e0 0x7f69e0145620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:18.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.314+0000 7f69deffd700 1 --2- 172.21.15.78:0/1683065398 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f69e006cd00 0x7f69e0147a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:18.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.314+0000 7f69df7fe700 1 -- 172.21.15.78:0/1683065398 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f69e006cd00 msgr2=0x7f69e0147a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:18.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.314+0000 7f69df7fe700 1 --2- 172.21.15.78:0/1683065398 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f69e006cd00 0x7f69e0147a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:18.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.314+0000 7f69df7fe700 1 -- 172.21.15.78:0/1683065398 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f69e014b7e0 msgr2=0x7f69e0145620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:18.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.314+0000 7f69df7fe700 1 --2- 172.21.15.78:0/1683065398 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f69e014b7e0 0x7f69e0145620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:18.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.314+0000 7f69df7fe700 1 -- 172.21.15.78:0/1683065398 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d8009a70 con 0x7f69e0154b90 2023-10-14T11:58:18.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.314+0000 7f69de7fc700 1 --2- 172.21.15.78:0/1683065398 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f69e014b7e0 0x7f69e0145620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:18.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.314+0000 7f69df7fe700 1 --2- 172.21.15.78:0/1683065398 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69e0154b90 0x7f69e0145b60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f69e0068210 tx=0x7f69d8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:18.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.314+0000 7f69c77fe700 1 -- 172.21.15.78:0/1683065398 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d8012430 con 0x7f69e0154b90 2023-10-14T11:58:18.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.314+0000 7f69c77fe700 1 -- 172.21.15.78:0/1683065398 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d8020070 con 0x7f69e0154b90 2023-10-14T11:58:18.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.314+0000 7f69c77fe700 1 -- 172.21.15.78:0/1683065398 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d8004280 con 0x7f69e0154b90 2023-10-14T11:58:18.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.314+0000 7f69e5647700 1 -- 172.21.15.78:0/1683065398 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69e0157620 con 0x7f69e0154b90 2023-10-14T11:58:18.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.314+0000 7f69e5647700 1 -- 172.21.15.78:0/1683065398 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69e0157be0 con 0x7f69e0154b90 2023-10-14T11:58:18.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.315+0000 7f69e5647700 1 -- 172.21.15.78:0/1683065398 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69e00a9a80 con 0x7f69e0154b90 2023-10-14T11:58:18.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.315+0000 7f69c77fe700 1 -- 172.21.15.78:0/1683065398 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f69d80043e0 con 0x7f69e0154b90 2023-10-14T11:58:18.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.316+0000 7f69c77fe700 1 --2- 172.21.15.78:0/1683065398 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f69cc06af00 0x7f69cc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:18.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.316+0000 7f69c77fe700 1 -- 172.21.15.78:0/1683065398 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f69d8022070 con 0x7f69e0154b90 2023-10-14T11:58:18.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.316+0000 7f69deffd700 1 --2- 172.21.15.78:0/1683065398 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f69cc06af00 0x7f69cc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:18.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.316+0000 7f69deffd700 1 --2- 172.21.15.78:0/1683065398 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f69cc06af00 0x7f69cc06d3c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f69d40011e0 tx=0x7f69d4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:18.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.320+0000 7f69c77fe700 1 -- 172.21.15.78:0/1683065398 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f69d80562f0 con 0x7f69e0154b90 2023-10-14T11:58:18.376 INFO:tasks.ceph.mds.a.smithi078.stderr:2023-10-14T11:58:18.374+0000 7f4c7ce49700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-14T11:58:18.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.554+0000 7f69e5647700 1 -- 172.21.15.78:0/1683065398 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69e0074b80 con 0x7f69e0154b90 2023-10-14T11:58:18.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.554+0000 7f69c77fe700 1 -- 172.21.15.78:0/1683065398 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4904 (secure 0 0 0) 0x7f69d8059910 con 0x7f69e0154b90 2023-10-14T11:58:18.557 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:18.557 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/353043244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":353043244},{"type":"v1","addr":"172.21.15.78:6836","nonce":353043244}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6834/3389817343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":3389817343},{"type":"v1","addr":"172.21.15.195:6834","nonce":3389817343}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4267,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/460176701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6833","nonce":460176701},{"type":"v1","addr":"172.21.15.195:6835","nonce":460176701}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T11:58:14.535101+0000","modified":"2023-10-14T11:58:17.559202+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/920300787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":920300787},{"type":"v1","addr":"172.21.15.78:6837","nonce":920300787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-14T11:58:18.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.556+0000 7f69e5647700 1 -- 172.21.15.78:0/1683065398 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f69cc06af00 msgr2=0x7f69cc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:18.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.556+0000 7f69e5647700 1 --2- 172.21.15.78:0/1683065398 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f69cc06af00 0x7f69cc06d3c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f69d40011e0 tx=0x7f69d4009040 comp rx=0 tx=0).stop 2023-10-14T11:58:18.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.556+0000 7f69e5647700 1 -- 172.21.15.78:0/1683065398 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69e0154b90 msgr2=0x7f69e0145b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:18.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.556+0000 7f69e5647700 1 --2- 172.21.15.78:0/1683065398 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69e0154b90 0x7f69e0145b60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f69e0068210 tx=0x7f69d8004ad0 comp rx=0 tx=0).stop 2023-10-14T11:58:18.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.556+0000 7f69e5647700 1 -- 172.21.15.78:0/1683065398 shutdown_connections 2023-10-14T11:58:18.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.556+0000 7f69e5647700 1 --2- 172.21.15.78:0/1683065398 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69e0154b90 0x7f69e0145b60 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:18.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.556+0000 7f69e5647700 1 --2- 172.21.15.78:0/1683065398 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f69e006cd00 0x7f69e0147a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:18.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.556+0000 7f69e5647700 1 --2- 172.21.15.78:0/1683065398 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f69cc06af00 0x7f69cc06d3c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:18.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.556+0000 7f69e5647700 1 --2- 172.21.15.78:0/1683065398 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f69e014b7e0 0x7f69e0145620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:18.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.556+0000 7f69e5647700 1 -- 172.21.15.78:0/1683065398 >> 172.21.15.78:0/1683065398 conn(0x7f69e00ca2c0 msgr2=0x7f69e006b440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:18.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.556+0000 7f69e5647700 1 -- 172.21.15.78:0/1683065398 shutdown_connections 2023-10-14T11:58:18.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.556+0000 7f69e5647700 1 -- 172.21.15.78:0/1683065398 wait complete. 2023-10-14T11:58:18.561 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 8 2023-10-14T11:58:18.570 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T11:58:18.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.687+0000 7ff7dcdef700 1 Processor -- start 2023-10-14T11:58:18.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.688+0000 7ff7dcdef700 1 -- start start 2023-10-14T11:58:18.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.688+0000 7ff7dcdef700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff7d806ab40 0x7ff7d806f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:18.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.688+0000 7ff7dcdef700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff7d80c0350 0x7ff7d813d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:18.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.688+0000 7ff7dcdef700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d806b630 con 0x7ff7d81554e0 2023-10-14T11:58:18.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.688+0000 7ff7dcdef700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7d80c49f0 con 0x7ff7d80c0350 2023-10-14T11:58:18.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.688+0000 7ff7dcdef700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d80726a0 con 0x7ff7d806ab40 2023-10-14T11:58:18.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.689+0000 7ff7d6d9d700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff7d80c0350 0x7ff7d813d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:18.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.689+0000 7ff7d5d9b700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7ff7d81554e0 0x7ff7d806af20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49346/0 (socket says 172.21.15.78:49346) 2023-10-14T11:58:18.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.689+0000 7ff7d5d9b700 1 -- 172.21.15.78:0/4172350960 learned_addr learned my addr 172.21.15.78:0/4172350960 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:18.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.689+0000 7ff7d659c700 1 --2- 172.21.15.78:0/4172350960 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff7d806ab40 0x7ff7d806f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:18.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.689+0000 7ff7d6d9d700 1 -- 172.21.15.78:0/4172350960 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff7d806ab40 msgr2=0x7ff7d806f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:18.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.689+0000 7ff7d6d9d700 1 --2- 172.21.15.78:0/4172350960 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff7d806ab40 0x7ff7d806f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:18.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.689+0000 7ff7d6d9d700 1 -- 172.21.15.78:0/4172350960 >> v1:172.21.15.78:6789/0 conn(0x7ff7d81554e0 legacy=0x7ff7d806af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:18.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.689+0000 7ff7d559a700 1 -- 172.21.15.78:0/4172350960 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3220852928 0 0) 0x7ff7d806b630 con 0x7ff7d81554e0 2023-10-14T11:58:18.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.689+0000 7ff7d6d9d700 1 -- 172.21.15.78:0/4172350960 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d8072980 con 0x7ff7d80c0350 2023-10-14T11:58:18.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.689+0000 7ff7d6d9d700 1 --2- 172.21.15.78:0/4172350960 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7d80c0350 0x7ff7d813d7b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff7d0009d90 tx=0x7ff7d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fa297edf8bf68f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:18.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.689+0000 7ff7d559a700 1 -- 172.21.15.78:0/4172350960 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d000e070 con 0x7ff7d80c0350 2023-10-14T11:58:18.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.689+0000 7ff7d559a700 1 -- 172.21.15.78:0/4172350960 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7d000bbd0 con 0x7ff7d80c0350 2023-10-14T11:58:18.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.689+0000 7ff7d559a700 1 -- 172.21.15.78:0/4172350960 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d0012430 con 0x7ff7d80c0350 2023-10-14T11:58:18.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.690+0000 7ff7dcdef700 1 -- 172.21.15.78:0/4172350960 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7d80c0350 msgr2=0x7ff7d813d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:18.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.690+0000 7ff7dcdef700 1 --2- 172.21.15.78:0/4172350960 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7d80c0350 0x7ff7d813d7b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff7d0009d90 tx=0x7ff7d000b330 comp rx=0 tx=0).stop 2023-10-14T11:58:18.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.690+0000 7ff7dcdef700 1 -- 172.21.15.78:0/4172350960 shutdown_connections 2023-10-14T11:58:18.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.690+0000 7ff7dcdef700 1 --2- 172.21.15.78:0/4172350960 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7d80c0350 0x7ff7d813d7b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:18.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.690+0000 7ff7dcdef700 1 --2- 172.21.15.78:0/4172350960 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff7d806ab40 0x7ff7d806f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:18.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.690+0000 7ff7dcdef700 1 -- 172.21.15.78:0/4172350960 >> 172.21.15.78:0/4172350960 conn(0x7ff7d80c3ac0 msgr2=0x7ff7d80c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:18.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.690+0000 7ff7dcdef700 1 -- 172.21.15.78:0/4172350960 shutdown_connections 2023-10-14T11:58:18.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.690+0000 7ff7dcdef700 1 -- 172.21.15.78:0/4172350960 wait complete. 2023-10-14T11:58:18.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.691+0000 7ff7dcdef700 1 Processor -- start 2023-10-14T11:58:18.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.691+0000 7ff7dcdef700 1 -- start start 2023-10-14T11:58:18.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.692+0000 7ff7dcdef700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff7d806ab40 0x7ff7d80c6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:18.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.692+0000 7ff7dcdef700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7d80c0350 0x7ff7d8157970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:18.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.692+0000 7ff7dcdef700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff7d81554e0 0x7ff7d8157eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:18.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.692+0000 7ff7dcdef700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d81388f0 con 0x7ff7d80c0350 2023-10-14T11:58:18.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.692+0000 7ff7dcdef700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d8133390 con 0x7ff7d81554e0 2023-10-14T11:58:18.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.692+0000 7ff7dcdef700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d81e9e30 con 0x7ff7d806ab40 2023-10-14T11:58:18.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.692+0000 7ff7d5d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7d80c0350 0x7ff7d8157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:18.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.692+0000 7ff7d5d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7d80c0350 0x7ff7d8157970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44478/0 (socket says 172.21.15.78:44478) 2023-10-14T11:58:18.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.692+0000 7ff7d5d9b700 1 -- 172.21.15.78:0/2474586081 learned_addr learned my addr 172.21.15.78:0/2474586081 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:18.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.692+0000 7ff7d6d9d700 1 --2- 172.21.15.78:0/2474586081 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff7d81554e0 0x7ff7d8157eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:18.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.692+0000 7ff7d659c700 1 --2- 172.21.15.78:0/2474586081 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff7d806ab40 0x7ff7d80c6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:18.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.692+0000 7ff7d5d9b700 1 -- 172.21.15.78:0/2474586081 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff7d806ab40 msgr2=0x7ff7d80c6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:18.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.692+0000 7ff7d5d9b700 1 --2- 172.21.15.78:0/2474586081 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff7d806ab40 0x7ff7d80c6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:18.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.692+0000 7ff7d5d9b700 1 -- 172.21.15.78:0/2474586081 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff7d81554e0 msgr2=0x7ff7d8157eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:18.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.692+0000 7ff7d5d9b700 1 --2- 172.21.15.78:0/2474586081 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff7d81554e0 0x7ff7d8157eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:18.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.692+0000 7ff7d5d9b700 1 -- 172.21.15.78:0/2474586081 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d0009a70 con 0x7ff7d80c0350 2023-10-14T11:58:18.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.692+0000 7ff7d6d9d700 1 --2- 172.21.15.78:0/2474586081 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff7d81554e0 0x7ff7d8157eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:18.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.692+0000 7ff7d5d9b700 1 --2- 172.21.15.78:0/2474586081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7d80c0350 0x7ff7d8157970 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc00c580 tx=0x7ff7cc00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:18.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.692+0000 7ff7c6ffd700 1 -- 172.21.15.78:0/2474586081 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc004c80 con 0x7ff7d80c0350 2023-10-14T11:58:18.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.692+0000 7ff7c6ffd700 1 -- 172.21.15.78:0/2474586081 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7cc004e00 con 0x7ff7d80c0350 2023-10-14T11:58:18.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.692+0000 7ff7dcdef700 1 -- 172.21.15.78:0/2474586081 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d81e9520 con 0x7ff7d80c0350 2023-10-14T11:58:18.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.692+0000 7ff7c6ffd700 1 -- 172.21.15.78:0/2474586081 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc007310 con 0x7ff7d80c0350 2023-10-14T11:58:18.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.692+0000 7ff7dcdef700 1 -- 172.21.15.78:0/2474586081 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d81e97e0 con 0x7ff7d80c0350 2023-10-14T11:58:18.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.693+0000 7ff7dcdef700 1 -- 172.21.15.78:0/2474586081 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d8074b80 con 0x7ff7d80c0350 2023-10-14T11:58:18.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.694+0000 7ff7c6ffd700 1 -- 172.21.15.78:0/2474586081 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff7cc007470 con 0x7ff7d80c0350 2023-10-14T11:58:18.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.694+0000 7ff7c6ffd700 1 --2- 172.21.15.78:0/2474586081 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff7b806af00 0x7ff7b806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:18.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.694+0000 7ff7c6ffd700 1 -- 172.21.15.78:0/2474586081 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff7cc0892c0 con 0x7ff7d80c0350 2023-10-14T11:58:18.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.694+0000 7ff7d659c700 1 --2- 172.21.15.78:0/2474586081 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff7b806af00 0x7ff7b806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:18.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.694+0000 7ff7d659c700 1 --2- 172.21.15.78:0/2474586081 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff7b806af00 0x7ff7b806d3c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff7d8156650 tx=0x7ff7c8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:18.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.699+0000 7ff7c6ffd700 1 -- 172.21.15.78:0/2474586081 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff7cc054400 con 0x7ff7d80c0350 2023-10-14T11:58:18.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.890+0000 7ff7dcdef700 1 -- 172.21.15.78:0/2474586081 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7d81e9f70 con 0x7ff7d80c0350 2023-10-14T11:58:18.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.891+0000 7ff7c6ffd700 1 -- 172.21.15.78:0/2474586081 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16481 (secure 0 0 0) 0x7ff7cc057a20 con 0x7ff7d80c0350 2023-10-14T11:58:18.893 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:18.894 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":21,"fsid":"4c5dc134-3092-406a-a1e1-13e6d9e926e8","created":"2023-10-14T11:57:57.363750+0000","modified":"2023-10-14T11:58:16.139413+0000","last_up_change":"2023-10-14T11:58:06.982472+0000","last_in_change":"2023-10-14T11:57:59.821373+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T11:58:09.001200+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T11:58:12.778573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T11:58:13.490286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"625a5fc0-9474-4eae-b80e-14c27a74ad19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6801","nonce":3884969780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6803","nonce":3884969780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6807","nonce":3884969780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6805","nonce":3884969780}]},"public_addr":"172.21.15.78:6801/3884969780","cluster_addr":"172.21.15.78:6803/3884969780","heartbeat_back_addr":"172.21.15.78:6807/3884969780","heartbeat_front_addr":"172.21.15.78:6805/3884969780","state":["exists","up"]},{"osd":1,"uuid":"49d63db1-2981-49ab-9c40-c50fc744b982","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6809","nonce":2176928406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6811","nonce":2176928406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6815","nonce":2176928406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6813","nonce":2176928406}]},"public_addr":"172.21.15.78:6809/2176928406","cluster_addr":"172.21.15.78:6811/2176928406","heartbeat_back_addr":"172.21.15.78:6815/2176928406","heartbeat_front_addr":"172.21.15.78:6813/2176928406","state":["exists","up"]},{"osd":2,"uuid":"1d7d0b29-cbd1-4717-9fe6-765a8d4998b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6825","nonce":3647044577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6827","nonce":3647044577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6831","nonce":3647044577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6829","nonce":3647044577}]},"public_addr":"172.21.15.78:6825/3647044577","cluster_addr":"172.21.15.78:6827/3647044577","heartbeat_back_addr":"172.21.15.78:6831/3647044577","heartbeat_front_addr":"172.21.15.78:6829/3647044577","state":["exists","up"]},{"osd":3,"uuid":"2f75ded1-e9e2-40e7-99ae-45e1e425e1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6817","nonce":3584112288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6819","nonce":3584112288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6823","nonce":3584112288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6821","nonce":3584112288}]},"public_addr":"172.21.15.78:6817/3584112288","cluster_addr":"172.21.15.78:6819/3584112288","heartbeat_back_addr":"172.21.15.78:6823/3584112288","heartbeat_front_addr":"172.21.15.78:6821/3584112288","state":["exists","up"]},{"osd":4,"uuid":"5130b432-8358-46cb-8a31-3064c976f6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6801","nonce":3732756301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6803","nonce":3732756301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6807","nonce":3732756301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6805","nonce":3732756301}]},"public_addr":"172.21.15.195:6801/3732756301","cluster_addr":"172.21.15.195:6803/3732756301","heartbeat_back_addr":"172.21.15.195:6807/3732756301","heartbeat_front_addr":"172.21.15.195:6805/3732756301","state":["exists","up"]},{"osd":5,"uuid":"60e234a9-a506-4e40-9c19-49a58f3cf28c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6825","nonce":2857706411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6827","nonce":2857706411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6831","nonce":2857706411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6829","nonce":2857706411}]},"public_addr":"172.21.15.195:6825/2857706411","cluster_addr":"172.21.15.195:6827/2857706411","heartbeat_back_addr":"172.21.15.195:6831/2857706411","heartbeat_front_addr":"172.21.15.195:6829/2857706411","state":["exists","up"]},{"osd":6,"uuid":"df7d2dd6-26b6-4eb1-96e5-94d1f3955b33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6810","nonce":72554374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6813","nonce":72554374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6820","nonce":72554374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6817","nonce":72554374}]},"public_addr":"172.21.15.195:6810/72554374","cluster_addr":"172.21.15.195:6813/72554374","heartbeat_back_addr":"172.21.15.195:6820/72554374","heartbeat_front_addr":"172.21.15.195:6817/72554374","state":["exists","up"]},{"osd":7,"uuid":"47a9d4de-f56d-4aa4-9900-c4a9cf9b2f97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6809","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6811","nonce":482472979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6816","nonce":482472979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6823","nonce":482472979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6821","nonce":482472979}]},"public_addr":"172.21.15.195:6811/482472979","cluster_addr":"172.21.15.195:6816/482472979","heartbeat_back_addr":"172.21.15.195:6823/482472979","heartbeat_front_addr":"172.21.15.195:6821/482472979","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.754003+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.005994+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.043603+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.024506+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.062585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.033767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.994825+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.038300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T11:58:18.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.893+0000 7ff7dcdef700 1 -- 172.21.15.78:0/2474586081 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff7b806af00 msgr2=0x7ff7b806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:18.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.893+0000 7ff7dcdef700 1 --2- 172.21.15.78:0/2474586081 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff7b806af00 0x7ff7b806d3c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff7d8156650 tx=0x7ff7c8008040 comp rx=0 tx=0).stop 2023-10-14T11:58:18.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.893+0000 7ff7dcdef700 1 -- 172.21.15.78:0/2474586081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7d80c0350 msgr2=0x7ff7d8157970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:18.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.893+0000 7ff7dcdef700 1 --2- 172.21.15.78:0/2474586081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7d80c0350 0x7ff7d8157970 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc00c580 tx=0x7ff7cc00c940 comp rx=0 tx=0).stop 2023-10-14T11:58:18.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.893+0000 7ff7dcdef700 1 -- 172.21.15.78:0/2474586081 shutdown_connections 2023-10-14T11:58:18.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.893+0000 7ff7dcdef700 1 --2- 172.21.15.78:0/2474586081 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff7b806af00 0x7ff7b806d3c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:18.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.893+0000 7ff7dcdef700 1 --2- 172.21.15.78:0/2474586081 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff7d81554e0 0x7ff7d8157eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:18.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.893+0000 7ff7dcdef700 1 --2- 172.21.15.78:0/2474586081 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff7d806ab40 0x7ff7d80c6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:18.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.893+0000 7ff7dcdef700 1 --2- 172.21.15.78:0/2474586081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7d80c0350 0x7ff7d8157970 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:18.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.893+0000 7ff7dcdef700 1 -- 172.21.15.78:0/2474586081 >> 172.21.15.78:0/2474586081 conn(0x7ff7d80c3ac0 msgr2=0x7ff7d8134ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:18.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.893+0000 7ff7dcdef700 1 -- 172.21.15.78:0/2474586081 shutdown_connections 2023-10-14T11:58:18.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:18.893+0000 7ff7dcdef700 1 -- 172.21.15.78:0/2474586081 wait complete. 2023-10-14T11:58:18.907 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-14T11:58:18.908 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T11:58:19.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.026+0000 7f92ff857700 1 Processor -- start 2023-10-14T11:58:19.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.026+0000 7f92ff857700 1 -- start start 2023-10-14T11:58:19.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.027+0000 7f92ff857700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f92f806b420 0x7f92f80c4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:19.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.027+0000 7f92ff857700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f80c86d0 con 0x7f92f80c8240 2023-10-14T11:58:19.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.027+0000 7f92ff857700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f80c8970 con 0x7f92f814b5b0 2023-10-14T11:58:19.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.027+0000 7f92ff857700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92f80c9040 con 0x7f92f806b420 2023-10-14T11:58:19.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.027+0000 7f92fcdf2700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f92f80c8240 0x7f92f8155580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49360/0 (socket says 172.21.15.78:49360) 2023-10-14T11:58:19.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.027+0000 7f92fcdf2700 1 -- 172.21.15.78:0/618286006 learned_addr learned my addr 172.21.15.78:0/618286006 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:19.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.027+0000 7f92fddf4700 1 --2- 172.21.15.78:0/618286006 >> v2:172.21.15.195:3300/0 conn(0x7f92f806b420 0x7f92f80c4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:19.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.027+0000 7f92effff700 1 -- 172.21.15.78:0/618286006 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2542323063 0 0) 0x7f92f80c86d0 con 0x7f92f80c8240 2023-10-14T11:58:19.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.027+0000 7f92effff700 1 -- 172.21.15.78:0/618286006 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92e40035b0 con 0x7f92f80c8240 2023-10-14T11:58:19.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.027+0000 7f92effff700 1 -- 172.21.15.78:0/618286006 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2508901975 0 0) 0x7f92e40035b0 con 0x7f92f80c8240 2023-10-14T11:58:19.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.027+0000 7f92effff700 1 -- 172.21.15.78:0/618286006 >> v2:172.21.15.195:3300/0 conn(0x7f92f806b420 msgr2=0x7f92f80c4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:19.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.027+0000 7f92effff700 1 --2- 172.21.15.78:0/618286006 >> v2:172.21.15.195:3300/0 conn(0x7f92f806b420 0x7f92f80c4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:19.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.027+0000 7f92effff700 1 -- 172.21.15.78:0/618286006 >> v1:172.21.15.195:6789/0 conn(0x7f92f814b5b0 legacy=0x7f92f80c9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:19.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.027+0000 7f92effff700 1 -- 172.21.15.78:0/618286006 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f80c9320 con 0x7f92f80c8240 2023-10-14T11:58:19.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.027+0000 7f92effff700 1 -- 172.21.15.78:0/618286006 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441799917 0 0) 0x7f92e8001670 con 0x7f92f80c8240 2023-10-14T11:58:19.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.027+0000 7f92effff700 1 -- 172.21.15.78:0/618286006 >> v1:172.21.15.78:6789/0 conn(0x7f92f80c8240 legacy=0x7f92f8155580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:19.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.027+0000 7f92effff700 1 --2- 172.21.15.78:0/618286006 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f92e4003a00 0x7f92e4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:19.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.027+0000 7f92effff700 1 --2- 172.21.15.78:0/618286006 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f92e4004850 0x7f92e4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:19.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.027+0000 7f92effff700 1 --2- 172.21.15.78:0/618286006 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92e4005210 0x7f92e400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:19.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.027+0000 7f92effff700 1 -- 172.21.15.78:0/618286006 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f80c9040 con 0x7f92e4005210 2023-10-14T11:58:19.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.027+0000 7f92effff700 1 -- 172.21.15.78:0/618286006 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e400c1b0 con 0x7f92e4004850 2023-10-14T11:58:19.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.027+0000 7f92effff700 1 -- 172.21.15.78:0/618286006 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92e400c370 con 0x7f92e4003a00 2023-10-14T11:58:19.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.028+0000 7f92fddf4700 1 --2- 172.21.15.78:0/618286006 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92e4005210 0x7f92e400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:19.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.028+0000 7f92fd5f3700 1 --2- 172.21.15.78:0/618286006 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f92e4003a00 0x7f92e4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:19.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.028+0000 7f92fcdf2700 1 --2- 172.21.15.78:0/618286006 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f92e4004850 0x7f92e4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:19.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.028+0000 7f92fddf4700 1 -- 172.21.15.78:0/618286006 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f92e4003a00 msgr2=0x7f92e4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:19.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.028+0000 7f92fddf4700 1 --2- 172.21.15.78:0/618286006 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f92e4003a00 0x7f92e4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:19.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.028+0000 7f92fddf4700 1 -- 172.21.15.78:0/618286006 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f92e4004850 msgr2=0x7f92e4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:19.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.028+0000 7f92fddf4700 1 --2- 172.21.15.78:0/618286006 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f92e4004850 0x7f92e4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:19.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.028+0000 7f92fddf4700 1 -- 172.21.15.78:0/618286006 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e40035b0 con 0x7f92e4005210 2023-10-14T11:58:19.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.028+0000 7f92fddf4700 1 -- 172.21.15.78:0/618286006 reap_dead start 2023-10-14T11:58:19.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.028+0000 7f92fddf4700 1 --2- 172.21.15.78:0/618286006 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92e4005210 0x7f92e400ba10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f92f0009fc0 tx=0x7f92e4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90474ea7b1ded274 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:19.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.028+0000 7f92effff700 1 -- 172.21.15.78:0/618286006 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f0003a20 con 0x7f92e4005210 2023-10-14T11:58:19.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.028+0000 7f92effff700 1 -- 172.21.15.78:0/618286006 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92f000f040 con 0x7f92e4005210 2023-10-14T11:58:19.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.028+0000 7f92effff700 1 -- 172.21.15.78:0/618286006 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f0003de0 con 0x7f92e4005210 2023-10-14T11:58:19.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.029+0000 7f92ff857700 1 -- 172.21.15.78:0/618286006 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92e4005210 msgr2=0x7f92e400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:19.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.029+0000 7f92ff857700 1 --2- 172.21.15.78:0/618286006 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92e4005210 0x7f92e400ba10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f92f0009fc0 tx=0x7f92e4005880 comp rx=0 tx=0).stop 2023-10-14T11:58:19.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.029+0000 7f92ff857700 1 -- 172.21.15.78:0/618286006 shutdown_connections 2023-10-14T11:58:19.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.029+0000 7f92ff857700 1 --2- 172.21.15.78:0/618286006 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92e4005210 0x7f92e400ba10 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:19.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.029+0000 7f92ff857700 1 -- 172.21.15.78:0/618286006 >> 172.21.15.78:0/618286006 conn(0x7f92f80abe30 msgr2=0x7f92f80ac240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:19.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.029+0000 7f92ff857700 1 -- 172.21.15.78:0/618286006 shutdown_connections 2023-10-14T11:58:19.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.029+0000 7f92ff857700 1 -- 172.21.15.78:0/618286006 wait complete. 2023-10-14T11:58:19.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.030+0000 7f92ff857700 1 Processor -- start 2023-10-14T11:58:19.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.030+0000 7f92ff857700 1 -- start start 2023-10-14T11:58:19.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.031+0000 7f92ff857700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f92e4005210 0x7f92f81de400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:19.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.031+0000 7f92ff857700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92f80c1880 0x7f92f81485a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:19.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.031+0000 7f92ff857700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f92f81dfb50 0x7f92f8148ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:19.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.031+0000 7f92ff857700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f81e9740 con 0x7f92f80c1880 2023-10-14T11:58:19.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.031+0000 7f92ff857700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f8149020 con 0x7f92e4005210 2023-10-14T11:58:19.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.031+0000 7f92ff857700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92f80ad170 con 0x7f92f81dfb50 2023-10-14T11:58:19.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.031+0000 7f92fcdf2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92f80c1880 0x7f92f81485a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:19.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.031+0000 7f92fcdf2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92f80c1880 0x7f92f81485a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44496/0 (socket says 172.21.15.78:44496) 2023-10-14T11:58:19.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.031+0000 7f92fcdf2700 1 -- 172.21.15.78:0/4234542161 learned_addr learned my addr 172.21.15.78:0/4234542161 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:19.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.031+0000 7f92fd5f3700 1 --2- 172.21.15.78:0/4234542161 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f92e4005210 0x7f92f81de400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:19.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.031+0000 7f92fddf4700 1 --2- 172.21.15.78:0/4234542161 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f92f81dfb50 0x7f92f8148ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:19.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.031+0000 7f92fcdf2700 1 -- 172.21.15.78:0/4234542161 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f92f81dfb50 msgr2=0x7f92f8148ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:19.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.031+0000 7f92fcdf2700 1 --2- 172.21.15.78:0/4234542161 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f92f81dfb50 0x7f92f8148ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:19.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.031+0000 7f92fcdf2700 1 -- 172.21.15.78:0/4234542161 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f92e4005210 msgr2=0x7f92f81de400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:19.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.031+0000 7f92fcdf2700 1 --2- 172.21.15.78:0/4234542161 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f92e4005210 0x7f92f81de400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:19.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.031+0000 7f92fcdf2700 1 -- 172.21.15.78:0/4234542161 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f80c2140 con 0x7f92f80c1880 2023-10-14T11:58:19.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.031+0000 7f92fcdf2700 1 --2- 172.21.15.78:0/4234542161 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92f80c1880 0x7f92f81485a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f92e8007290 tx=0x7f92e80075a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:19.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.031+0000 7f92edffb700 1 -- 172.21.15.78:0/4234542161 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e8011580 con 0x7f92f80c1880 2023-10-14T11:58:19.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.031+0000 7f92edffb700 1 -- 172.21.15.78:0/4234542161 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92e80116e0 con 0x7f92f80c1880 2023-10-14T11:58:19.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.031+0000 7f92ff857700 1 -- 172.21.15.78:0/4234542161 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92f81eda10 con 0x7f92f80c1880 2023-10-14T11:58:19.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.031+0000 7f92edffb700 1 -- 172.21.15.78:0/4234542161 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e8011a90 con 0x7f92f80c1880 2023-10-14T11:58:19.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.031+0000 7f92ff857700 1 -- 172.21.15.78:0/4234542161 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92f80ad660 con 0x7f92f80c1880 2023-10-14T11:58:19.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.032+0000 7f92ff857700 1 -- 172.21.15.78:0/4234542161 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92f81e9890 con 0x7f92f80c1880 2023-10-14T11:58:19.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.033+0000 7f92edffb700 1 -- 172.21.15.78:0/4234542161 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f92e8011bf0 con 0x7f92f80c1880 2023-10-14T11:58:19.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.033+0000 7f92edffb700 1 --2- 172.21.15.78:0/4234542161 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f92dc06af00 0x7f92dc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:19.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.033+0000 7f92edffb700 1 -- 172.21.15.78:0/4234542161 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f92e80896a0 con 0x7f92f80c1880 2023-10-14T11:58:19.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.033+0000 7f92fd5f3700 1 --2- 172.21.15.78:0/4234542161 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f92dc06af00 0x7f92dc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:19.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.033+0000 7f92fd5f3700 1 --2- 172.21.15.78:0/4234542161 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f92dc06af00 0x7f92dc06d3c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f92f4000c00 tx=0x7f92f4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:19.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.037+0000 7f92edffb700 1 -- 172.21.15.78:0/4234542161 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f92f81e9890 con 0x7f92f80c1880 2023-10-14T11:58:19.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.224+0000 7f92ff857700 1 -- 172.21.15.78:0/4234542161 --> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f92f80c8c50 con 0x7f92dc06af00 2023-10-14T11:58:19.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.231+0000 7f92edffb700 1 -- 172.21.15.78:0/4234542161 <== mgr.4104 v2:172.21.15.78:6832/3171890074 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264833 (secure 0 0 0) 0x7f92f4001ca0 con 0x7f92dc06af00 2023-10-14T11:58:19.236 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:19.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.234+0000 7f92ff857700 1 -- 172.21.15.78:0/4234542161 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f92dc06af00 msgr2=0x7f92dc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:19.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.234+0000 7f92ff857700 1 --2- 172.21.15.78:0/4234542161 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f92dc06af00 0x7f92dc06d3c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f92f4000c00 tx=0x7f92f4009040 comp rx=0 tx=0).stop 2023-10-14T11:58:19.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.234+0000 7f92ff857700 1 -- 172.21.15.78:0/4234542161 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92f80c1880 msgr2=0x7f92f81485a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:19.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.234+0000 7f92ff857700 1 --2- 172.21.15.78:0/4234542161 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92f80c1880 0x7f92f81485a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f92e8007290 tx=0x7f92e80075a0 comp rx=0 tx=0).stop 2023-10-14T11:58:19.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.234+0000 7f92ff857700 1 -- 172.21.15.78:0/4234542161 shutdown_connections 2023-10-14T11:58:19.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.234+0000 7f92ff857700 1 --2- 172.21.15.78:0/4234542161 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f92f81dfb50 0x7f92f8148ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:19.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.234+0000 7f92ff857700 1 --2- 172.21.15.78:0/4234542161 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f92dc06af00 0x7f92dc06d3c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:19.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.234+0000 7f92ff857700 1 --2- 172.21.15.78:0/4234542161 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f92e4005210 0x7f92f81de400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:19.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.234+0000 7f92ff857700 1 --2- 172.21.15.78:0/4234542161 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92f80c1880 0x7f92f81485a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:19.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.234+0000 7f92ff857700 1 -- 172.21.15.78:0/4234542161 >> 172.21.15.78:0/4234542161 conn(0x7f92f80abe30 msgr2=0x7f92f814a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:19.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.235+0000 7f92ff857700 1 -- 172.21.15.78:0/4234542161 shutdown_connections 2023-10-14T11:58:19.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.235+0000 7f92ff857700 1 -- 172.21.15.78:0/4234542161 wait complete. 2023-10-14T11:58:19.241 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2023-10-14T11:58:19.248 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2023-10-14T11:58:18.958920+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218064,"kb_used_data":1876,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756656,"statfs":{"total":773094113280,"available":772870815744,"internally_reserved":0,"allocated":1921024,"data_stored":1783808,"data_compressed":65244,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.831539"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297796+0000","last_change":"2023-10-14T11:58:15.161449+0000","last_active":"2023-10-14T11:58:16.297796+0000","last_peered":"2023-10-14T11:58:16.297796+0000","last_clean":"2023-10-14T11:58:16.297796+0000","last_became_active":"2023-10-14T11:58:15.161100+0000","last_became_peered":"2023-10-14T11:58:15.161100+0000","last_unstale":"2023-10-14T11:58:16.297796+0000","last_undegraded":"2023-10-14T11:58:16.297796+0000","last_fullsized":"2023-10-14T11:58:16.297796+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:47:58.533238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.145107+0000","last_change":"2023-10-14T11:58:15.158210+0000","last_active":"2023-10-14T11:58:16.145107+0000","last_peered":"2023-10-14T11:58:16.145107+0000","last_clean":"2023-10-14T11:58:16.145107+0000","last_became_active":"2023-10-14T11:58:15.157886+0000","last_became_peered":"2023-10-14T11:58:15.157886+0000","last_unstale":"2023-10-14T11:58:16.145107+0000","last_undegraded":"2023-10-14T11:58:16.145107+0000","last_fullsized":"2023-10-14T11:58:16.145107+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:23:17.281352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296226+0000","last_change":"2023-10-14T11:58:15.161880+0000","last_active":"2023-10-14T11:58:16.296226+0000","last_peered":"2023-10-14T11:58:16.296226+0000","last_clean":"2023-10-14T11:58:16.296226+0000","last_became_active":"2023-10-14T11:58:15.161603+0000","last_became_peered":"2023-10-14T11:58:15.161603+0000","last_unstale":"2023-10-14T11:58:16.296226+0000","last_undegraded":"2023-10-14T11:58:16.296226+0000","last_fullsized":"2023-10-14T11:58:16.296226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:01:03.783116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296854+0000","last_change":"2023-10-14T11:58:15.167778+0000","last_active":"2023-10-14T11:58:16.296854+0000","last_peered":"2023-10-14T11:58:16.296854+0000","last_clean":"2023-10-14T11:58:16.296854+0000","last_became_active":"2023-10-14T11:58:15.167471+0000","last_became_peered":"2023-10-14T11:58:15.167471+0000","last_unstale":"2023-10-14T11:58:16.296854+0000","last_undegraded":"2023-10-14T11:58:16.296854+0000","last_fullsized":"2023-10-14T11:58:16.296854+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:07:58.884850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296653+0000","last_change":"2023-10-14T11:58:15.160788+0000","last_active":"2023-10-14T11:58:16.296653+0000","last_peered":"2023-10-14T11:58:16.296653+0000","last_clean":"2023-10-14T11:58:16.296653+0000","last_became_active":"2023-10-14T11:58:15.160525+0000","last_became_peered":"2023-10-14T11:58:15.160525+0000","last_unstale":"2023-10-14T11:58:16.296653+0000","last_undegraded":"2023-10-14T11:58:16.296653+0000","last_fullsized":"2023-10-14T11:58:16.296653+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:35:58.025887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297023+0000","last_change":"2023-10-14T11:58:15.167657+0000","last_active":"2023-10-14T11:58:16.297023+0000","last_peered":"2023-10-14T11:58:16.297023+0000","last_clean":"2023-10-14T11:58:16.297023+0000","last_became_active":"2023-10-14T11:58:15.167383+0000","last_became_peered":"2023-10-14T11:58:15.167383+0000","last_unstale":"2023-10-14T11:58:16.297023+0000","last_undegraded":"2023-10-14T11:58:16.297023+0000","last_fullsized":"2023-10-14T11:58:16.297023+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:53:19.926796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296834+0000","last_change":"2023-10-14T11:58:15.156244+0000","last_active":"2023-10-14T11:58:16.296834+0000","last_peered":"2023-10-14T11:58:16.296834+0000","last_clean":"2023-10-14T11:58:16.296834+0000","last_became_active":"2023-10-14T11:58:15.155621+0000","last_became_peered":"2023-10-14T11:58:15.155621+0000","last_unstale":"2023-10-14T11:58:16.296834+0000","last_undegraded":"2023-10-14T11:58:16.296834+0000","last_fullsized":"2023-10-14T11:58:16.296834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:07:58.884850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298542+0000","last_change":"2023-10-14T11:58:15.165865+0000","last_active":"2023-10-14T11:58:16.298542+0000","last_peered":"2023-10-14T11:58:16.298542+0000","last_clean":"2023-10-14T11:58:16.298542+0000","last_became_active":"2023-10-14T11:58:15.165613+0000","last_became_peered":"2023-10-14T11:58:15.165613+0000","last_unstale":"2023-10-14T11:58:16.298542+0000","last_undegraded":"2023-10-14T11:58:16.298542+0000","last_fullsized":"2023-10-14T11:58:16.298542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:32:38.145488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297728+0000","last_change":"2023-10-14T11:58:15.167047+0000","last_active":"2023-10-14T11:58:16.297728+0000","last_peered":"2023-10-14T11:58:16.297728+0000","last_clean":"2023-10-14T11:58:16.297728+0000","last_became_active":"2023-10-14T11:58:15.166362+0000","last_became_peered":"2023-10-14T11:58:15.166362+0000","last_unstale":"2023-10-14T11:58:16.297728+0000","last_undegraded":"2023-10-14T11:58:16.297728+0000","last_fullsized":"2023-10-14T11:58:16.297728+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:39:27.703491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144707+0000","last_change":"2023-10-14T11:58:15.164412+0000","last_active":"2023-10-14T11:58:16.144707+0000","last_peered":"2023-10-14T11:58:16.144707+0000","last_clean":"2023-10-14T11:58:16.144707+0000","last_became_active":"2023-10-14T11:58:15.164187+0000","last_became_peered":"2023-10-14T11:58:15.164187+0000","last_unstale":"2023-10-14T11:58:16.144707+0000","last_undegraded":"2023-10-14T11:58:16.144707+0000","last_fullsized":"2023-10-14T11:58:16.144707+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:59:03.464936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144466+0000","last_change":"2023-10-14T11:58:15.163857+0000","last_active":"2023-10-14T11:58:16.144466+0000","last_peered":"2023-10-14T11:58:16.144466+0000","last_clean":"2023-10-14T11:58:16.144466+0000","last_became_active":"2023-10-14T11:58:15.163575+0000","last_became_peered":"2023-10-14T11:58:15.163575+0000","last_unstale":"2023-10-14T11:58:16.144466+0000","last_undegraded":"2023-10-14T11:58:16.144466+0000","last_fullsized":"2023-10-14T11:58:16.144466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:01:03.783116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144222+0000","last_change":"2023-10-14T11:58:15.149569+0000","last_active":"2023-10-14T11:58:16.144222+0000","last_peered":"2023-10-14T11:58:16.144222+0000","last_clean":"2023-10-14T11:58:16.144222+0000","last_became_active":"2023-10-14T11:58:15.149161+0000","last_became_peered":"2023-10-14T11:58:15.149161+0000","last_unstale":"2023-10-14T11:58:16.144222+0000","last_undegraded":"2023-10-14T11:58:16.144222+0000","last_fullsized":"2023-10-14T11:58:16.144222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:19:30.143095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.299094+0000","last_change":"2023-10-14T11:58:15.147173+0000","last_active":"2023-10-14T11:58:16.299094+0000","last_peered":"2023-10-14T11:58:16.299094+0000","last_clean":"2023-10-14T11:58:16.299094+0000","last_became_active":"2023-10-14T11:58:15.146910+0000","last_became_peered":"2023-10-14T11:58:15.146910+0000","last_unstale":"2023-10-14T11:58:16.299094+0000","last_undegraded":"2023-10-14T11:58:16.299094+0000","last_fullsized":"2023-10-14T11:58:16.299094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:59:03.464936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296040+0000","last_change":"2023-10-14T11:58:15.169906+0000","last_active":"2023-10-14T11:58:16.296040+0000","last_peered":"2023-10-14T11:58:16.296040+0000","last_clean":"2023-10-14T11:58:16.296040+0000","last_became_active":"2023-10-14T11:58:15.169660+0000","last_became_peered":"2023-10-14T11:58:15.169660+0000","last_unstale":"2023-10-14T11:58:16.296040+0000","last_undegraded":"2023-10-14T11:58:16.296040+0000","last_fullsized":"2023-10-14T11:58:16.296040+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:32:38.145488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144820+0000","last_change":"2023-10-14T11:58:15.159353+0000","last_active":"2023-10-14T11:58:16.144820+0000","last_peered":"2023-10-14T11:58:16.144820+0000","last_clean":"2023-10-14T11:58:16.144820+0000","last_became_active":"2023-10-14T11:58:15.159095+0000","last_became_peered":"2023-10-14T11:58:15.159095+0000","last_unstale":"2023-10-14T11:58:16.144820+0000","last_undegraded":"2023-10-14T11:58:16.144820+0000","last_fullsized":"2023-10-14T11:58:16.144820+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:33:00.861183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296349+0000","last_change":"2023-10-14T11:58:15.169061+0000","last_active":"2023-10-14T11:58:16.296349+0000","last_peered":"2023-10-14T11:58:16.296349+0000","last_clean":"2023-10-14T11:58:16.296349+0000","last_became_active":"2023-10-14T11:58:15.168627+0000","last_became_peered":"2023-10-14T11:58:15.168627+0000","last_unstale":"2023-10-14T11:58:16.296349+0000","last_undegraded":"2023-10-14T11:58:16.296349+0000","last_fullsized":"2023-10-14T11:58:16.296349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:23:17.281352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297017+0000","last_change":"2023-10-14T11:58:15.162773+0000","last_active":"2023-10-14T11:58:16.297017+0000","last_peered":"2023-10-14T11:58:16.297017+0000","last_clean":"2023-10-14T11:58:16.297017+0000","last_became_active":"2023-10-14T11:58:15.162514+0000","last_became_peered":"2023-10-14T11:58:15.162514+0000","last_unstale":"2023-10-14T11:58:16.297017+0000","last_undegraded":"2023-10-14T11:58:16.297017+0000","last_fullsized":"2023-10-14T11:58:16.297017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:02:52.001714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298805+0000","last_change":"2023-10-14T11:58:15.164541+0000","last_active":"2023-10-14T11:58:16.298805+0000","last_peered":"2023-10-14T11:58:16.298805+0000","last_clean":"2023-10-14T11:58:16.298805+0000","last_became_active":"2023-10-14T11:58:15.164309+0000","last_became_peered":"2023-10-14T11:58:15.164309+0000","last_unstale":"2023-10-14T11:58:16.298805+0000","last_undegraded":"2023-10-14T11:58:16.298805+0000","last_fullsized":"2023-10-14T11:58:16.298805+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:22:04.803877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296451+0000","last_change":"2023-10-14T11:58:15.162171+0000","last_active":"2023-10-14T11:58:16.296451+0000","last_peered":"2023-10-14T11:58:16.296451+0000","last_clean":"2023-10-14T11:58:16.296451+0000","last_became_active":"2023-10-14T11:58:15.161894+0000","last_became_peered":"2023-10-14T11:58:15.161894+0000","last_unstale":"2023-10-14T11:58:16.296451+0000","last_undegraded":"2023-10-14T11:58:16.296451+0000","last_fullsized":"2023-10-14T11:58:16.296451+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:15:07.231894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296298+0000","last_change":"2023-10-14T11:58:14.145266+0000","last_active":"2023-10-14T11:58:16.296298+0000","last_peered":"2023-10-14T11:58:16.296298+0000","last_clean":"2023-10-14T11:58:16.296298+0000","last_became_active":"2023-10-14T11:58:14.144885+0000","last_became_peered":"2023-10-14T11:58:14.144885+0000","last_unstale":"2023-10-14T11:58:16.296298+0000","last_undegraded":"2023-10-14T11:58:16.296298+0000","last_fullsized":"2023-10-14T11:58:16.296298+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:06:27.985367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.299066+0000","last_change":"2023-10-14T11:58:15.148017+0000","last_active":"2023-10-14T11:58:16.299066+0000","last_peered":"2023-10-14T11:58:16.299066+0000","last_clean":"2023-10-14T11:58:16.299066+0000","last_became_active":"2023-10-14T11:58:15.147796+0000","last_became_peered":"2023-10-14T11:58:15.147796+0000","last_unstale":"2023-10-14T11:58:16.299066+0000","last_undegraded":"2023-10-14T11:58:16.299066+0000","last_fullsized":"2023-10-14T11:58:16.299066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:23:17.281352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.299129+0000","last_change":"2023-10-14T11:58:14.152335+0000","last_active":"2023-10-14T11:58:16.299129+0000","last_peered":"2023-10-14T11:58:16.299129+0000","last_clean":"2023-10-14T11:58:16.299129+0000","last_became_active":"2023-10-14T11:58:14.152064+0000","last_became_peered":"2023-10-14T11:58:14.152064+0000","last_unstale":"2023-10-14T11:58:16.299129+0000","last_undegraded":"2023-10-14T11:58:16.299129+0000","last_fullsized":"2023-10-14T11:58:16.299129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:21:22.993387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296225+0000","last_change":"2023-10-14T11:58:15.169510+0000","last_active":"2023-10-14T11:58:16.296225+0000","last_peered":"2023-10-14T11:58:16.296225+0000","last_clean":"2023-10-14T11:58:16.296225+0000","last_became_active":"2023-10-14T11:58:15.166488+0000","last_became_peered":"2023-10-14T11:58:15.166488+0000","last_unstale":"2023-10-14T11:58:16.296225+0000","last_undegraded":"2023-10-14T11:58:16.296225+0000","last_fullsized":"2023-10-14T11:58:16.296225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:23:17.281352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297447+0000","last_change":"2023-10-14T11:58:14.150333+0000","last_active":"2023-10-14T11:58:16.297447+0000","last_peered":"2023-10-14T11:58:16.297447+0000","last_clean":"2023-10-14T11:58:16.297447+0000","last_became_active":"2023-10-14T11:58:14.149979+0000","last_became_peered":"2023-10-14T11:58:14.149979+0000","last_unstale":"2023-10-14T11:58:16.297447+0000","last_undegraded":"2023-10-14T11:58:16.297447+0000","last_fullsized":"2023-10-14T11:58:16.297447+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:23:16.277717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297176+0000","last_change":"2023-10-14T11:58:15.163307+0000","last_active":"2023-10-14T11:58:16.297176+0000","last_peered":"2023-10-14T11:58:16.297176+0000","last_clean":"2023-10-14T11:58:16.297176+0000","last_became_active":"2023-10-14T11:58:15.163015+0000","last_became_peered":"2023-10-14T11:58:15.163015+0000","last_unstale":"2023-10-14T11:58:16.297176+0000","last_undegraded":"2023-10-14T11:58:16.297176+0000","last_fullsized":"2023-10-14T11:58:16.297176+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:21:10.166149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296924+0000","last_change":"2023-10-14T11:58:14.150931+0000","last_active":"2023-10-14T11:58:16.296924+0000","last_peered":"2023-10-14T11:58:16.296924+0000","last_clean":"2023-10-14T11:58:16.296924+0000","last_became_active":"2023-10-14T11:58:14.150629+0000","last_became_peered":"2023-10-14T11:58:14.150629+0000","last_unstale":"2023-10-14T11:58:16.296924+0000","last_undegraded":"2023-10-14T11:58:16.296924+0000","last_fullsized":"2023-10-14T11:58:16.296924+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:22:03.800242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297270+0000","last_change":"2023-10-14T11:58:15.163629+0000","last_active":"2023-10-14T11:58:16.297270+0000","last_peered":"2023-10-14T11:58:16.297270+0000","last_clean":"2023-10-14T11:58:16.297270+0000","last_became_active":"2023-10-14T11:58:15.163043+0000","last_became_peered":"2023-10-14T11:58:15.163043+0000","last_unstale":"2023-10-14T11:58:16.297270+0000","last_undegraded":"2023-10-14T11:58:16.297270+0000","last_fullsized":"2023-10-14T11:58:16.297270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:39:27.703491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298703+0000","last_change":"2023-10-14T11:58:14.154507+0000","last_active":"2023-10-14T11:58:16.298703+0000","last_peered":"2023-10-14T11:58:16.298703+0000","last_clean":"2023-10-14T11:58:16.298703+0000","last_became_active":"2023-10-14T11:58:14.154202+0000","last_became_peered":"2023-10-14T11:58:14.154202+0000","last_unstale":"2023-10-14T11:58:16.298703+0000","last_undegraded":"2023-10-14T11:58:16.298703+0000","last_fullsized":"2023-10-14T11:58:16.298703+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:59:45.791176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.143513+0000","last_change":"2023-10-14T11:58:15.158639+0000","last_active":"2023-10-14T11:58:16.143513+0000","last_peered":"2023-10-14T11:58:16.143513+0000","last_clean":"2023-10-14T11:58:16.143513+0000","last_became_active":"2023-10-14T11:58:15.158402+0000","last_became_peered":"2023-10-14T11:58:15.158402+0000","last_unstale":"2023-10-14T11:58:16.143513+0000","last_undegraded":"2023-10-14T11:58:16.143513+0000","last_fullsized":"2023-10-14T11:58:16.143513+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:21:23.997022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144263+0000","last_change":"2023-10-14T11:58:14.139502+0000","last_active":"2023-10-14T11:58:16.144263+0000","last_peered":"2023-10-14T11:58:16.144263+0000","last_clean":"2023-10-14T11:58:16.144263+0000","last_became_active":"2023-10-14T11:58:14.139053+0000","last_became_peered":"2023-10-14T11:58:14.139053+0000","last_unstale":"2023-10-14T11:58:16.144263+0000","last_undegraded":"2023-10-14T11:58:16.144263+0000","last_fullsized":"2023-10-14T11:58:16.144263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:10:10.291507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296162+0000","last_change":"2023-10-14T11:58:15.169858+0000","last_active":"2023-10-14T11:58:16.296162+0000","last_peered":"2023-10-14T11:58:16.296162+0000","last_clean":"2023-10-14T11:58:16.296162+0000","last_became_active":"2023-10-14T11:58:15.166724+0000","last_became_peered":"2023-10-14T11:58:15.166724+0000","last_unstale":"2023-10-14T11:58:16.296162+0000","last_undegraded":"2023-10-14T11:58:16.296162+0000","last_fullsized":"2023-10-14T11:58:16.296162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:45:35.707554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.299064+0000","last_change":"2023-10-14T11:58:14.154253+0000","last_active":"2023-10-14T11:58:16.299064+0000","last_peered":"2023-10-14T11:58:16.299064+0000","last_clean":"2023-10-14T11:58:16.299064+0000","last_became_active":"2023-10-14T11:58:14.152400+0000","last_became_peered":"2023-10-14T11:58:14.152400+0000","last_unstale":"2023-10-14T11:58:16.299064+0000","last_undegraded":"2023-10-14T11:58:16.299064+0000","last_fullsized":"2023-10-14T11:58:16.299064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:03:15.889637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298935+0000","last_change":"2023-10-14T11:58:15.167532+0000","last_active":"2023-10-14T11:58:16.298935+0000","last_peered":"2023-10-14T11:58:16.298935+0000","last_clean":"2023-10-14T11:58:16.298935+0000","last_became_active":"2023-10-14T11:58:15.167270+0000","last_became_peered":"2023-10-14T11:58:15.167270+0000","last_unstale":"2023-10-14T11:58:16.298935+0000","last_undegraded":"2023-10-14T11:58:16.298935+0000","last_fullsized":"2023-10-14T11:58:16.298935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:33:00.861183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296354+0000","last_change":"2023-10-14T11:58:14.147092+0000","last_active":"2023-10-14T11:58:16.296354+0000","last_peered":"2023-10-14T11:58:16.296354+0000","last_clean":"2023-10-14T11:58:16.296354+0000","last_became_active":"2023-10-14T11:58:14.146377+0000","last_became_peered":"2023-10-14T11:58:14.146377+0000","last_unstale":"2023-10-14T11:58:16.296354+0000","last_undegraded":"2023-10-14T11:58:16.296354+0000","last_fullsized":"2023-10-14T11:58:16.296354+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:25:37.999389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144924+0000","last_change":"2023-10-14T11:58:15.160286+0000","last_active":"2023-10-14T11:58:16.144924+0000","last_peered":"2023-10-14T11:58:16.144924+0000","last_clean":"2023-10-14T11:58:16.144924+0000","last_became_active":"2023-10-14T11:58:15.159308+0000","last_became_peered":"2023-10-14T11:58:15.159308+0000","last_unstale":"2023-10-14T11:58:16.144924+0000","last_undegraded":"2023-10-14T11:58:16.144924+0000","last_fullsized":"2023-10-14T11:58:16.144924+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:45:35.707554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144601+0000","last_change":"2023-10-14T11:58:14.148448+0000","last_active":"2023-10-14T11:58:16.144601+0000","last_peered":"2023-10-14T11:58:16.144601+0000","last_clean":"2023-10-14T11:58:16.144601+0000","last_became_active":"2023-10-14T11:58:14.148155+0000","last_became_peered":"2023-10-14T11:58:14.148155+0000","last_unstale":"2023-10-14T11:58:16.144601+0000","last_undegraded":"2023-10-14T11:58:16.144601+0000","last_fullsized":"2023-10-14T11:58:16.144601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:19:29.139460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296785+0000","last_change":"2023-10-14T11:58:15.166089+0000","last_active":"2023-10-14T11:58:16.296785+0000","last_peered":"2023-10-14T11:58:16.296785+0000","last_clean":"2023-10-14T11:58:16.296785+0000","last_became_active":"2023-10-14T11:58:15.164889+0000","last_became_peered":"2023-10-14T11:58:15.164889+0000","last_unstale":"2023-10-14T11:58:16.296785+0000","last_undegraded":"2023-10-14T11:58:16.296785+0000","last_fullsized":"2023-10-14T11:58:16.296785+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:15.482221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296897+0000","last_change":"2023-10-14T11:58:14.150455+0000","last_active":"2023-10-14T11:58:16.296897+0000","last_peered":"2023-10-14T11:58:16.296897+0000","last_clean":"2023-10-14T11:58:16.296897+0000","last_became_active":"2023-10-14T11:58:14.150195+0000","last_became_peered":"2023-10-14T11:58:14.150195+0000","last_unstale":"2023-10-14T11:58:16.296897+0000","last_undegraded":"2023-10-14T11:58:16.296897+0000","last_fullsized":"2023-10-14T11:58:16.296897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:32:59.857548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296229+0000","last_change":"2023-10-14T11:58:15.170459+0000","last_active":"2023-10-14T11:58:16.296229+0000","last_peered":"2023-10-14T11:58:16.296229+0000","last_clean":"2023-10-14T11:58:16.296229+0000","last_became_active":"2023-10-14T11:58:15.170096+0000","last_became_peered":"2023-10-14T11:58:15.170096+0000","last_unstale":"2023-10-14T11:58:16.296229+0000","last_undegraded":"2023-10-14T11:58:16.296229+0000","last_fullsized":"2023-10-14T11:58:16.296229+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:21:15.443882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.145168+0000","last_change":"2023-10-14T11:58:14.147259+0000","last_active":"2023-10-14T11:58:16.145168+0000","last_peered":"2023-10-14T11:58:16.145168+0000","last_clean":"2023-10-14T11:58:16.145168+0000","last_became_active":"2023-10-14T11:58:14.146982+0000","last_became_peered":"2023-10-14T11:58:14.146982+0000","last_unstale":"2023-10-14T11:58:16.145168+0000","last_undegraded":"2023-10-14T11:58:16.145168+0000","last_fullsized":"2023-10-14T11:58:16.145168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:22:03.800242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144175+0000","last_change":"2023-10-14T11:58:15.158917+0000","last_active":"2023-10-14T11:58:16.144175+0000","last_peered":"2023-10-14T11:58:16.144175+0000","last_clean":"2023-10-14T11:58:16.144175+0000","last_became_active":"2023-10-14T11:58:15.158578+0000","last_became_peered":"2023-10-14T11:58:15.158578+0000","last_unstale":"2023-10-14T11:58:16.144175+0000","last_undegraded":"2023-10-14T11:58:16.144175+0000","last_fullsized":"2023-10-14T11:58:16.144175+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:06:28.989002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144380+0000","last_change":"2023-10-14T11:58:14.152849+0000","last_active":"2023-10-14T11:58:16.144380+0000","last_peered":"2023-10-14T11:58:16.144380+0000","last_clean":"2023-10-14T11:58:16.144380+0000","last_became_active":"2023-10-14T11:58:14.152544+0000","last_became_peered":"2023-10-14T11:58:14.152544+0000","last_unstale":"2023-10-14T11:58:16.144380+0000","last_undegraded":"2023-10-14T11:58:16.144380+0000","last_fullsized":"2023-10-14T11:58:16.144380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:30:50.303297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297109+0000","last_change":"2023-10-14T11:58:15.160605+0000","last_active":"2023-10-14T11:58:16.297109+0000","last_peered":"2023-10-14T11:58:16.297109+0000","last_clean":"2023-10-14T11:58:16.297109+0000","last_became_active":"2023-10-14T11:58:15.160323+0000","last_became_peered":"2023-10-14T11:58:15.160323+0000","last_unstale":"2023-10-14T11:58:16.297109+0000","last_undegraded":"2023-10-14T11:58:16.297109+0000","last_fullsized":"2023-10-14T11:58:16.297109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:53:19.926796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296162+0000","last_change":"2023-10-14T11:58:14.145911+0000","last_active":"2023-10-14T11:58:16.296162+0000","last_peered":"2023-10-14T11:58:16.296162+0000","last_clean":"2023-10-14T11:58:16.296162+0000","last_became_active":"2023-10-14T11:58:14.144322+0000","last_became_peered":"2023-10-14T11:58:14.144322+0000","last_unstale":"2023-10-14T11:58:16.296162+0000","last_undegraded":"2023-10-14T11:58:16.296162+0000","last_fullsized":"2023-10-14T11:58:16.296162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:32:03.667630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.295965+0000","last_change":"2023-10-14T11:58:15.165189+0000","last_active":"2023-10-14T11:58:16.295965+0000","last_peered":"2023-10-14T11:58:16.295965+0000","last_clean":"2023-10-14T11:58:16.295965+0000","last_became_active":"2023-10-14T11:58:15.164911+0000","last_became_peered":"2023-10-14T11:58:15.164911+0000","last_unstale":"2023-10-14T11:58:16.295965+0000","last_undegraded":"2023-10-14T11:58:16.295965+0000","last_fullsized":"2023-10-14T11:58:16.295965+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:10:17.502466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297270+0000","last_change":"2023-10-14T11:58:14.152736+0000","last_active":"2023-10-14T11:58:16.297270+0000","last_peered":"2023-10-14T11:58:16.297270+0000","last_clean":"2023-10-14T11:58:16.297270+0000","last_became_active":"2023-10-14T11:58:14.152401+0000","last_became_peered":"2023-10-14T11:58:14.152401+0000","last_unstale":"2023-10-14T11:58:16.297270+0000","last_undegraded":"2023-10-14T11:58:16.297270+0000","last_fullsized":"2023-10-14T11:58:16.297270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:01:02.779481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296948+0000","last_change":"2023-10-14T11:58:15.155365+0000","last_active":"2023-10-14T11:58:16.296948+0000","last_peered":"2023-10-14T11:58:16.296948+0000","last_clean":"2023-10-14T11:58:16.296948+0000","last_became_active":"2023-10-14T11:58:15.155102+0000","last_became_peered":"2023-10-14T11:58:15.155102+0000","last_unstale":"2023-10-14T11:58:16.296948+0000","last_undegraded":"2023-10-14T11:58:16.296948+0000","last_fullsized":"2023-10-14T11:58:16.296948+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:02:52.001714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.143938+0000","last_change":"2023-10-14T11:58:14.148862+0000","last_active":"2023-10-14T11:58:16.143938+0000","last_peered":"2023-10-14T11:58:16.143938+0000","last_clean":"2023-10-14T11:58:16.143938+0000","last_became_active":"2023-10-14T11:58:14.148409+0000","last_became_peered":"2023-10-14T11:58:14.148409+0000","last_unstale":"2023-10-14T11:58:16.143938+0000","last_undegraded":"2023-10-14T11:58:16.143938+0000","last_fullsized":"2023-10-14T11:58:16.143938+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:06:27.985367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.143490+0000","last_change":"2023-10-14T11:58:15.156428+0000","last_active":"2023-10-14T11:58:16.143490+0000","last_peered":"2023-10-14T11:58:16.143490+0000","last_clean":"2023-10-14T11:58:16.143490+0000","last_became_active":"2023-10-14T11:58:15.155818+0000","last_became_peered":"2023-10-14T11:58:15.155818+0000","last_unstale":"2023-10-14T11:58:16.143490+0000","last_undegraded":"2023-10-14T11:58:16.143490+0000","last_fullsized":"2023-10-14T11:58:16.143490+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:33:00.861183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298391+0000","last_change":"2023-10-14T11:58:14.149643+0000","last_active":"2023-10-14T11:58:16.298391+0000","last_peered":"2023-10-14T11:58:16.298391+0000","last_clean":"2023-10-14T11:58:16.298391+0000","last_became_active":"2023-10-14T11:58:14.149330+0000","last_became_peered":"2023-10-14T11:58:14.149330+0000","last_unstale":"2023-10-14T11:58:16.298391+0000","last_undegraded":"2023-10-14T11:58:16.298391+0000","last_fullsized":"2023-10-14T11:58:16.298391+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:14:56.067093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296718+0000","last_change":"2023-10-14T11:58:15.156412+0000","last_active":"2023-10-14T11:58:16.296718+0000","last_peered":"2023-10-14T11:58:16.296718+0000","last_clean":"2023-10-14T11:58:16.296718+0000","last_became_active":"2023-10-14T11:58:15.155951+0000","last_became_peered":"2023-10-14T11:58:15.155951+0000","last_unstale":"2023-10-14T11:58:16.296718+0000","last_undegraded":"2023-10-14T11:58:16.296718+0000","last_fullsized":"2023-10-14T11:58:16.296718+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:15.482221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.143832+0000","last_change":"2023-10-14T11:58:14.135648+0000","last_active":"2023-10-14T11:58:16.143832+0000","last_peered":"2023-10-14T11:58:16.143832+0000","last_clean":"2023-10-14T11:58:16.143832+0000","last_became_active":"2023-10-14T11:58:14.135104+0000","last_became_peered":"2023-10-14T11:58:14.135104+0000","last_unstale":"2023-10-14T11:58:16.143832+0000","last_undegraded":"2023-10-14T11:58:16.143832+0000","last_fullsized":"2023-10-14T11:58:16.143832+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:47:41.184305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297156+0000","last_change":"2023-10-14T11:58:15.157354+0000","last_active":"2023-10-14T11:58:16.297156+0000","last_peered":"2023-10-14T11:58:16.297156+0000","last_clean":"2023-10-14T11:58:16.297156+0000","last_became_active":"2023-10-14T11:58:15.157084+0000","last_became_peered":"2023-10-14T11:58:15.157084+0000","last_unstale":"2023-10-14T11:58:16.297156+0000","last_undegraded":"2023-10-14T11:58:16.297156+0000","last_fullsized":"2023-10-14T11:58:16.297156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:32:40.379634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144935+0000","last_change":"2023-10-14T11:58:14.148246+0000","last_active":"2023-10-14T11:58:16.144935+0000","last_peered":"2023-10-14T11:58:16.144935+0000","last_clean":"2023-10-14T11:58:16.144935+0000","last_became_active":"2023-10-14T11:58:14.147943+0000","last_became_peered":"2023-10-14T11:58:14.147943+0000","last_unstale":"2023-10-14T11:58:16.144935+0000","last_undegraded":"2023-10-14T11:58:16.144935+0000","last_fullsized":"2023-10-14T11:58:16.144935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:19:29.139460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297125+0000","last_change":"2023-10-14T11:58:15.162066+0000","last_active":"2023-10-14T11:58:16.297125+0000","last_peered":"2023-10-14T11:58:16.297125+0000","last_clean":"2023-10-14T11:58:16.297125+0000","last_became_active":"2023-10-14T11:58:15.161798+0000","last_became_peered":"2023-10-14T11:58:15.161798+0000","last_unstale":"2023-10-14T11:58:16.297125+0000","last_undegraded":"2023-10-14T11:58:16.297125+0000","last_fullsized":"2023-10-14T11:58:16.297125+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:32:40.379634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297269+0000","last_change":"2023-10-14T11:58:14.147024+0000","last_active":"2023-10-14T11:58:16.297269+0000","last_peered":"2023-10-14T11:58:16.297269+0000","last_clean":"2023-10-14T11:58:16.297269+0000","last_became_active":"2023-10-14T11:58:14.146675+0000","last_became_peered":"2023-10-14T11:58:14.146675+0000","last_unstale":"2023-10-14T11:58:16.297269+0000","last_undegraded":"2023-10-14T11:58:16.297269+0000","last_fullsized":"2023-10-14T11:58:16.297269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:45:34.703919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.143832+0000","last_change":"2023-10-14T11:58:15.157978+0000","last_active":"2023-10-14T11:58:16.143832+0000","last_peered":"2023-10-14T11:58:16.143832+0000","last_clean":"2023-10-14T11:58:16.143832+0000","last_became_active":"2023-10-14T11:58:15.157698+0000","last_became_peered":"2023-10-14T11:58:15.157698+0000","last_unstale":"2023-10-14T11:58:16.143832+0000","last_undegraded":"2023-10-14T11:58:16.143832+0000","last_fullsized":"2023-10-14T11:58:16.143832+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:10:11.295142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144794+0000","last_change":"2023-10-14T11:58:15.164985+0000","last_active":"2023-10-14T11:58:16.144794+0000","last_peered":"2023-10-14T11:58:16.144794+0000","last_clean":"2023-10-14T11:58:16.144794+0000","last_became_active":"2023-10-14T11:58:15.164355+0000","last_became_peered":"2023-10-14T11:58:15.164355+0000","last_unstale":"2023-10-14T11:58:16.144794+0000","last_undegraded":"2023-10-14T11:58:16.144794+0000","last_fullsized":"2023-10-14T11:58:16.144794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:25:39.003024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298382+0000","last_change":"2023-10-14T11:58:14.153483+0000","last_active":"2023-10-14T11:58:16.298382+0000","last_peered":"2023-10-14T11:58:16.298382+0000","last_clean":"2023-10-14T11:58:16.298382+0000","last_became_active":"2023-10-14T11:58:14.153174+0000","last_became_peered":"2023-10-14T11:58:14.153174+0000","last_unstale":"2023-10-14T11:58:16.298382+0000","last_undegraded":"2023-10-14T11:58:16.298382+0000","last_fullsized":"2023-10-14T11:58:16.298382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:06:27.985367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298697+0000","last_change":"2023-10-14T11:58:15.171141+0000","last_active":"2023-10-14T11:58:16.298697+0000","last_peered":"2023-10-14T11:58:16.298697+0000","last_clean":"2023-10-14T11:58:16.298697+0000","last_became_active":"2023-10-14T11:58:15.170853+0000","last_became_peered":"2023-10-14T11:58:15.170853+0000","last_unstale":"2023-10-14T11:58:16.298697+0000","last_undegraded":"2023-10-14T11:58:16.298697+0000","last_fullsized":"2023-10-14T11:58:16.298697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:45:35.707554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296962+0000","last_change":"2023-10-14T11:58:14.153660+0000","last_active":"2023-10-14T11:58:16.296962+0000","last_peered":"2023-10-14T11:58:16.296962+0000","last_clean":"2023-10-14T11:58:16.296962+0000","last_became_active":"2023-10-14T11:58:14.153419+0000","last_became_peered":"2023-10-14T11:58:14.153419+0000","last_unstale":"2023-10-14T11:58:16.296962+0000","last_undegraded":"2023-10-14T11:58:16.296962+0000","last_fullsized":"2023-10-14T11:58:16.296962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:06:27.985367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T11:58:15.149401+0000","last_change":"2023-10-14T11:58:10.616726+0000","last_active":"2023-10-14T11:58:15.149401+0000","last_peered":"2023-10-14T11:58:15.149401+0000","last_clean":"2023-10-14T11:58:15.149401+0000","last_became_active":"2023-10-14T11:58:10.615628+0000","last_became_peered":"2023-10-14T11:58:10.615628+0000","last_unstale":"2023-10-14T11:58:15.149401+0000","last_undegraded":"2023-10-14T11:58:15.149401+0000","last_fullsized":"2023-10-14T11:58:15.149401+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:09.001265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:09.001265+0000","last_clean_scrub_stamp":"2023-10-14T11:58:09.001265+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:22:50.496050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.295927+0000","last_change":"2023-10-14T11:58:15.173632+0000","last_active":"2023-10-14T11:58:16.295927+0000","last_peered":"2023-10-14T11:58:16.295927+0000","last_clean":"2023-10-14T11:58:16.295927+0000","last_became_active":"2023-10-14T11:58:15.172996+0000","last_became_peered":"2023-10-14T11:58:15.172996+0000","last_unstale":"2023-10-14T11:58:16.295927+0000","last_undegraded":"2023-10-14T11:58:16.295927+0000","last_fullsized":"2023-10-14T11:58:16.295927+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:22:04.803877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296824+0000","last_change":"2023-10-14T11:58:14.150078+0000","last_active":"2023-10-14T11:58:16.296824+0000","last_peered":"2023-10-14T11:58:16.296824+0000","last_clean":"2023-10-14T11:58:16.296824+0000","last_became_active":"2023-10-14T11:58:14.149788+0000","last_became_peered":"2023-10-14T11:58:14.149788+0000","last_unstale":"2023-10-14T11:58:16.296824+0000","last_undegraded":"2023-10-14T11:58:16.296824+0000","last_fullsized":"2023-10-14T11:58:16.296824+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:10:10.291507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297353+0000","last_change":"2023-10-14T11:58:14.151386+0000","last_active":"2023-10-14T11:58:16.297353+0000","last_peered":"2023-10-14T11:58:16.297353+0000","last_clean":"2023-10-14T11:58:16.297353+0000","last_became_active":"2023-10-14T11:58:14.151041+0000","last_became_peered":"2023-10-14T11:58:14.151041+0000","last_unstale":"2023-10-14T11:58:16.297353+0000","last_undegraded":"2023-10-14T11:58:16.297353+0000","last_fullsized":"2023-10-14T11:58:16.297353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:21:14.440247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144942+0000","last_change":"2023-10-14T11:58:15.168958+0000","last_active":"2023-10-14T11:58:16.144942+0000","last_peered":"2023-10-14T11:58:16.144942+0000","last_clean":"2023-10-14T11:58:16.144942+0000","last_became_active":"2023-10-14T11:58:15.164516+0000","last_became_peered":"2023-10-14T11:58:15.164516+0000","last_unstale":"2023-10-14T11:58:16.144942+0000","last_undegraded":"2023-10-14T11:58:16.144942+0000","last_fullsized":"2023-10-14T11:58:16.144942+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:01:03.783116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144989+0000","last_change":"2023-10-14T11:58:14.148617+0000","last_active":"2023-10-14T11:58:16.144989+0000","last_peered":"2023-10-14T11:58:16.144989+0000","last_clean":"2023-10-14T11:58:16.144989+0000","last_became_active":"2023-10-14T11:58:14.148324+0000","last_became_peered":"2023-10-14T11:58:14.148324+0000","last_unstale":"2023-10-14T11:58:16.144989+0000","last_undegraded":"2023-10-14T11:58:16.144989+0000","last_fullsized":"2023-10-14T11:58:16.144989+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:10:16.498831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.299219+0000","last_change":"2023-10-14T11:58:15.162441+0000","last_active":"2023-10-14T11:58:16.299219+0000","last_peered":"2023-10-14T11:58:16.299219+0000","last_clean":"2023-10-14T11:58:16.299219+0000","last_became_active":"2023-10-14T11:58:15.162047+0000","last_became_peered":"2023-10-14T11:58:15.162047+0000","last_unstale":"2023-10-14T11:58:16.299219+0000","last_undegraded":"2023-10-14T11:58:16.299219+0000","last_fullsized":"2023-10-14T11:58:16.299219+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:15.482221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144522+0000","last_change":"2023-10-14T11:58:14.151993+0000","last_active":"2023-10-14T11:58:16.144522+0000","last_peered":"2023-10-14T11:58:16.144522+0000","last_clean":"2023-10-14T11:58:16.144522+0000","last_became_active":"2023-10-14T11:58:14.151721+0000","last_became_peered":"2023-10-14T11:58:14.151721+0000","last_unstale":"2023-10-14T11:58:16.144522+0000","last_undegraded":"2023-10-14T11:58:16.144522+0000","last_fullsized":"2023-10-14T11:58:16.144522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:21:22.993387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296156+0000","last_change":"2023-10-14T11:58:15.170037+0000","last_active":"2023-10-14T11:58:16.296156+0000","last_peered":"2023-10-14T11:58:16.296156+0000","last_clean":"2023-10-14T11:58:16.296156+0000","last_became_active":"2023-10-14T11:58:15.169745+0000","last_became_peered":"2023-10-14T11:58:15.169745+0000","last_unstale":"2023-10-14T11:58:16.296156+0000","last_undegraded":"2023-10-14T11:58:16.296156+0000","last_fullsized":"2023-10-14T11:58:16.296156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:33:00.861183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296261+0000","last_change":"2023-10-14T11:58:14.151181+0000","last_active":"2023-10-14T11:58:16.296261+0000","last_peered":"2023-10-14T11:58:16.296261+0000","last_clean":"2023-10-14T11:58:16.296261+0000","last_became_active":"2023-10-14T11:58:14.150859+0000","last_became_peered":"2023-10-14T11:58:14.150859+0000","last_unstale":"2023-10-14T11:58:16.296261+0000","last_undegraded":"2023-10-14T11:58:16.296261+0000","last_fullsized":"2023-10-14T11:58:16.296261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:21:22.993387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144028+0000","last_change":"2023-10-14T11:58:15.168925+0000","last_active":"2023-10-14T11:58:16.144028+0000","last_peered":"2023-10-14T11:58:16.144028+0000","last_clean":"2023-10-14T11:58:16.144028+0000","last_became_active":"2023-10-14T11:58:15.168534+0000","last_became_peered":"2023-10-14T11:58:15.168534+0000","last_unstale":"2023-10-14T11:58:16.144028+0000","last_undegraded":"2023-10-14T11:58:16.144028+0000","last_fullsized":"2023-10-14T11:58:16.144028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:58:44.511366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144781+0000","last_change":"2023-10-14T11:58:14.150372+0000","last_active":"2023-10-14T11:58:16.144781+0000","last_peered":"2023-10-14T11:58:16.144781+0000","last_clean":"2023-10-14T11:58:16.144781+0000","last_became_active":"2023-10-14T11:58:14.150095+0000","last_became_peered":"2023-10-14T11:58:14.150095+0000","last_unstale":"2023-10-14T11:58:16.144781+0000","last_undegraded":"2023-10-14T11:58:16.144781+0000","last_fullsized":"2023-10-14T11:58:16.144781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:32:37.141853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298587+0000","last_change":"2023-10-14T11:58:15.158813+0000","last_active":"2023-10-14T11:58:16.298587+0000","last_peered":"2023-10-14T11:58:16.298587+0000","last_clean":"2023-10-14T11:58:16.298587+0000","last_became_active":"2023-10-14T11:58:15.158536+0000","last_became_peered":"2023-10-14T11:58:15.158536+0000","last_unstale":"2023-10-14T11:58:16.298587+0000","last_undegraded":"2023-10-14T11:58:16.298587+0000","last_fullsized":"2023-10-14T11:58:16.298587+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:25:39.003024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.145036+0000","last_change":"2023-10-14T11:58:14.153044+0000","last_active":"2023-10-14T11:58:16.145036+0000","last_peered":"2023-10-14T11:58:16.145036+0000","last_clean":"2023-10-14T11:58:16.145036+0000","last_became_active":"2023-10-14T11:58:14.152746+0000","last_became_peered":"2023-10-14T11:58:14.152746+0000","last_unstale":"2023-10-14T11:58:16.145036+0000","last_undegraded":"2023-10-14T11:58:16.145036+0000","last_fullsized":"2023-10-14T11:58:16.145036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:30:50.303297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298842+0000","last_change":"2023-10-14T11:58:15.174434+0000","last_active":"2023-10-14T11:58:16.298842+0000","last_peered":"2023-10-14T11:58:16.298842+0000","last_clean":"2023-10-14T11:58:16.298842+0000","last_became_active":"2023-10-14T11:58:15.174141+0000","last_became_peered":"2023-10-14T11:58:15.174141+0000","last_unstale":"2023-10-14T11:58:16.298842+0000","last_undegraded":"2023-10-14T11:58:16.298842+0000","last_fullsized":"2023-10-14T11:58:16.298842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:21:15.443882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296762+0000","last_change":"2023-10-14T11:58:14.149244+0000","last_active":"2023-10-14T11:58:16.296762+0000","last_peered":"2023-10-14T11:58:16.296762+0000","last_clean":"2023-10-14T11:58:16.296762+0000","last_became_active":"2023-10-14T11:58:14.148960+0000","last_became_peered":"2023-10-14T11:58:14.148960+0000","last_unstale":"2023-10-14T11:58:16.296762+0000","last_undegraded":"2023-10-14T11:58:16.296762+0000","last_fullsized":"2023-10-14T11:58:16.296762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:32:03.667630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144154+0000","last_change":"2023-10-14T11:58:15.150214+0000","last_active":"2023-10-14T11:58:16.144154+0000","last_peered":"2023-10-14T11:58:16.144154+0000","last_clean":"2023-10-14T11:58:16.144154+0000","last_became_active":"2023-10-14T11:58:15.149927+0000","last_became_peered":"2023-10-14T11:58:15.149927+0000","last_unstale":"2023-10-14T11:58:16.144154+0000","last_undegraded":"2023-10-14T11:58:16.144154+0000","last_fullsized":"2023-10-14T11:58:16.144154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:45:35.707554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298684+0000","last_change":"2023-10-14T11:58:14.153886+0000","last_active":"2023-10-14T11:58:16.298684+0000","last_peered":"2023-10-14T11:58:16.298684+0000","last_clean":"2023-10-14T11:58:16.298684+0000","last_became_active":"2023-10-14T11:58:14.153365+0000","last_became_peered":"2023-10-14T11:58:14.153365+0000","last_unstale":"2023-10-14T11:58:16.298684+0000","last_undegraded":"2023-10-14T11:58:16.298684+0000","last_fullsized":"2023-10-14T11:58:16.298684+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:21:22.993387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297257+0000","last_change":"2023-10-14T11:58:15.168448+0000","last_active":"2023-10-14T11:58:16.297257+0000","last_peered":"2023-10-14T11:58:16.297257+0000","last_clean":"2023-10-14T11:58:16.297257+0000","last_became_active":"2023-10-14T11:58:15.167757+0000","last_became_peered":"2023-10-14T11:58:15.167757+0000","last_unstale":"2023-10-14T11:58:16.297257+0000","last_undegraded":"2023-10-14T11:58:16.297257+0000","last_fullsized":"2023-10-14T11:58:16.297257+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:21:10.166149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.145150+0000","last_change":"2023-10-14T11:58:14.145505+0000","last_active":"2023-10-14T11:58:16.145150+0000","last_peered":"2023-10-14T11:58:16.145150+0000","last_clean":"2023-10-14T11:58:16.145150+0000","last_became_active":"2023-10-14T11:58:14.145218+0000","last_became_peered":"2023-10-14T11:58:14.145218+0000","last_unstale":"2023-10-14T11:58:16.145150+0000","last_undegraded":"2023-10-14T11:58:16.145150+0000","last_fullsized":"2023-10-14T11:58:16.145150+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:32:03.667630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297202+0000","last_change":"2023-10-14T11:58:15.162290+0000","last_active":"2023-10-14T11:58:16.297202+0000","last_peered":"2023-10-14T11:58:16.297202+0000","last_clean":"2023-10-14T11:58:16.297202+0000","last_became_active":"2023-10-14T11:58:15.161987+0000","last_became_peered":"2023-10-14T11:58:15.161987+0000","last_unstale":"2023-10-14T11:58:16.297202+0000","last_undegraded":"2023-10-14T11:58:16.297202+0000","last_fullsized":"2023-10-14T11:58:16.297202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:42:47.990050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298895+0000","last_change":"2023-10-14T11:58:14.154431+0000","last_active":"2023-10-14T11:58:16.298895+0000","last_peered":"2023-10-14T11:58:16.298895+0000","last_clean":"2023-10-14T11:58:16.298895+0000","last_became_active":"2023-10-14T11:58:14.153698+0000","last_became_peered":"2023-10-14T11:58:14.153698+0000","last_unstale":"2023-10-14T11:58:16.298895+0000","last_undegraded":"2023-10-14T11:58:16.298895+0000","last_fullsized":"2023-10-14T11:58:16.298895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:10:10.291507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297698+0000","last_change":"2023-10-14T11:58:15.160591+0000","last_active":"2023-10-14T11:58:16.297698+0000","last_peered":"2023-10-14T11:58:16.297698+0000","last_clean":"2023-10-14T11:58:16.297698+0000","last_became_active":"2023-10-14T11:58:15.160280+0000","last_became_peered":"2023-10-14T11:58:15.160280+0000","last_unstale":"2023-10-14T11:58:16.297698+0000","last_undegraded":"2023-10-14T11:58:16.297698+0000","last_fullsized":"2023-10-14T11:58:16.297698+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:42:47.990050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.143334+0000","last_change":"2023-10-14T11:58:15.156633+0000","last_active":"2023-10-14T11:58:16.143334+0000","last_peered":"2023-10-14T11:58:16.143334+0000","last_clean":"2023-10-14T11:58:16.143334+0000","last_became_active":"2023-10-14T11:58:15.156176+0000","last_became_peered":"2023-10-14T11:58:15.156176+0000","last_unstale":"2023-10-14T11:58:16.143334+0000","last_undegraded":"2023-10-14T11:58:16.143334+0000","last_fullsized":"2023-10-14T11:58:16.143334+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:25:39.003024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298424+0000","last_change":"2023-10-14T11:58:15.165873+0000","last_active":"2023-10-14T11:58:16.298424+0000","last_peered":"2023-10-14T11:58:16.298424+0000","last_clean":"2023-10-14T11:58:16.298424+0000","last_became_active":"2023-10-14T11:58:15.165592+0000","last_became_peered":"2023-10-14T11:58:15.165592+0000","last_unstale":"2023-10-14T11:58:16.298424+0000","last_undegraded":"2023-10-14T11:58:16.298424+0000","last_fullsized":"2023-10-14T11:58:16.298424+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:32:04.671265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296749+0000","last_change":"2023-10-14T11:58:15.165467+0000","last_active":"2023-10-14T11:58:16.296749+0000","last_peered":"2023-10-14T11:58:16.296749+0000","last_clean":"2023-10-14T11:58:16.296749+0000","last_became_active":"2023-10-14T11:58:15.165032+0000","last_became_peered":"2023-10-14T11:58:15.165032+0000","last_unstale":"2023-10-14T11:58:16.296749+0000","last_undegraded":"2023-10-14T11:58:16.296749+0000","last_fullsized":"2023-10-14T11:58:16.296749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:35:58.025887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296067+0000","last_change":"2023-10-14T11:58:15.170357+0000","last_active":"2023-10-14T11:58:16.296067+0000","last_peered":"2023-10-14T11:58:16.296067+0000","last_clean":"2023-10-14T11:58:16.296067+0000","last_became_active":"2023-10-14T11:58:15.170062+0000","last_became_peered":"2023-10-14T11:58:15.170062+0000","last_unstale":"2023-10-14T11:58:16.296067+0000","last_undegraded":"2023-10-14T11:58:16.296067+0000","last_fullsized":"2023-10-14T11:58:16.296067+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:25:39.003024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.299070+0000","last_change":"2023-10-14T11:58:15.156230+0000","last_active":"2023-10-14T11:58:16.299070+0000","last_peered":"2023-10-14T11:58:16.299070+0000","last_clean":"2023-10-14T11:58:16.299070+0000","last_became_active":"2023-10-14T11:58:15.155970+0000","last_became_peered":"2023-10-14T11:58:15.155970+0000","last_unstale":"2023-10-14T11:58:16.299070+0000","last_undegraded":"2023-10-14T11:58:16.299070+0000","last_fullsized":"2023-10-14T11:58:16.299070+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:15:07.231894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.145083+0000","last_change":"2023-10-14T11:58:15.158350+0000","last_active":"2023-10-14T11:58:16.145083+0000","last_peered":"2023-10-14T11:58:16.145083+0000","last_clean":"2023-10-14T11:58:16.145083+0000","last_became_active":"2023-10-14T11:58:15.158036+0000","last_became_peered":"2023-10-14T11:58:15.158036+0000","last_unstale":"2023-10-14T11:58:16.145083+0000","last_undegraded":"2023-10-14T11:58:16.145083+0000","last_fullsized":"2023-10-14T11:58:16.145083+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:21:15.443882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144612+0000","last_change":"2023-10-14T11:58:15.160864+0000","last_active":"2023-10-14T11:58:16.144612+0000","last_peered":"2023-10-14T11:58:16.144612+0000","last_clean":"2023-10-14T11:58:16.144612+0000","last_became_active":"2023-10-14T11:58:15.157887+0000","last_became_peered":"2023-10-14T11:58:15.157887+0000","last_unstale":"2023-10-14T11:58:16.144612+0000","last_undegraded":"2023-10-14T11:58:16.144612+0000","last_fullsized":"2023-10-14T11:58:16.144612+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:23:17.281352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.299205+0000","last_change":"2023-10-14T11:58:15.148543+0000","last_active":"2023-10-14T11:58:16.299205+0000","last_peered":"2023-10-14T11:58:16.299205+0000","last_clean":"2023-10-14T11:58:16.299205+0000","last_became_active":"2023-10-14T11:58:15.148273+0000","last_became_peered":"2023-10-14T11:58:15.148273+0000","last_unstale":"2023-10-14T11:58:16.299205+0000","last_undegraded":"2023-10-14T11:58:16.299205+0000","last_fullsized":"2023-10-14T11:58:16.299205+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:35:58.025887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296512+0000","last_change":"2023-10-14T11:58:15.162159+0000","last_active":"2023-10-14T11:58:16.296512+0000","last_peered":"2023-10-14T11:58:16.296512+0000","last_clean":"2023-10-14T11:58:16.296512+0000","last_became_active":"2023-10-14T11:58:15.161859+0000","last_became_peered":"2023-10-14T11:58:15.161859+0000","last_unstale":"2023-10-14T11:58:16.296512+0000","last_undegraded":"2023-10-14T11:58:16.296512+0000","last_fullsized":"2023-10-14T11:58:16.296512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:59:03.464936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298805+0000","last_change":"2023-10-14T11:58:15.147368+0000","last_active":"2023-10-14T11:58:16.298805+0000","last_peered":"2023-10-14T11:58:16.298805+0000","last_clean":"2023-10-14T11:58:16.298805+0000","last_became_active":"2023-10-14T11:58:15.147083+0000","last_became_peered":"2023-10-14T11:58:15.147083+0000","last_unstale":"2023-10-14T11:58:16.298805+0000","last_undegraded":"2023-10-14T11:58:16.298805+0000","last_fullsized":"2023-10-14T11:58:16.298805+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:01:03.783116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296598+0000","last_change":"2023-10-14T11:58:15.170122+0000","last_active":"2023-10-14T11:58:16.296598+0000","last_peered":"2023-10-14T11:58:16.296598+0000","last_clean":"2023-10-14T11:58:16.296598+0000","last_became_active":"2023-10-14T11:58:15.169483+0000","last_became_peered":"2023-10-14T11:58:15.169483+0000","last_unstale":"2023-10-14T11:58:16.296598+0000","last_undegraded":"2023-10-14T11:58:16.296598+0000","last_fullsized":"2023-10-14T11:58:16.296598+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:15:07.231894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296034+0000","last_change":"2023-10-14T11:58:15.170506+0000","last_active":"2023-10-14T11:58:16.296034+0000","last_peered":"2023-10-14T11:58:16.296034+0000","last_clean":"2023-10-14T11:58:16.296034+0000","last_became_active":"2023-10-14T11:58:15.170243+0000","last_became_peered":"2023-10-14T11:58:15.170243+0000","last_unstale":"2023-10-14T11:58:16.296034+0000","last_undegraded":"2023-10-14T11:58:16.296034+0000","last_fullsized":"2023-10-14T11:58:16.296034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:30:51.306932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144532+0000","last_change":"2023-10-14T11:58:15.147517+0000","last_active":"2023-10-14T11:58:16.144532+0000","last_peered":"2023-10-14T11:58:16.144532+0000","last_clean":"2023-10-14T11:58:16.144532+0000","last_became_active":"2023-10-14T11:58:15.147245+0000","last_became_peered":"2023-10-14T11:58:15.147245+0000","last_unstale":"2023-10-14T11:58:16.144532+0000","last_undegraded":"2023-10-14T11:58:16.144532+0000","last_fullsized":"2023-10-14T11:58:16.144532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:21:15.443882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65244,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":286720,"data_stored":666519,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95799999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":75729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":666575,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65600000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":69269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27900000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78500000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0900000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1439999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T11:58:19.255 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T11:58:19.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.409+0000 7f13ff258700 1 Processor -- start 2023-10-14T11:58:19.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.409+0000 7f13ff258700 1 -- start start 2023-10-14T11:58:19.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.410+0000 7f13ff258700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f13f814b5b0 0x7f13f80c8240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:19.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.410+0000 7f13ff258700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f13f806b420 0x7f13f80c9b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:19.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.410+0000 7f13ff258700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f13f8155580 0x7f13f80c4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.410+0000 7f13ff258700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13f80c8780 con 0x7f13f814b5b0 2023-10-14T11:58:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.410+0000 7f13ff258700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13f80ca060 con 0x7f13f806b420 2023-10-14T11:58:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.410+0000 7f13ff258700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13f80c8970 con 0x7f13f8155580 2023-10-14T11:58:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.410+0000 7f13fcff4700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f13f814b5b0 0x7f13f80c8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.410+0000 7f13fcff4700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f13f814b5b0 0x7f13f80c8240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44502/0 (socket says 172.21.15.78:44502) 2023-10-14T11:58:19.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.410+0000 7f13fcff4700 1 -- 172.21.15.78:0/798075198 learned_addr learned my addr 172.21.15.78:0/798075198 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:19.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.410+0000 7f13f7fff700 1 --2- 172.21.15.78:0/798075198 >> v2:172.21.15.195:3300/0 conn(0x7f13f806b420 0x7f13f80c9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:19.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.410+0000 7f13fd7f5700 1 --2- 172.21.15.78:0/798075198 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f13f8155580 0x7f13f80c4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:19.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.410+0000 7f13fcff4700 1 -- 172.21.15.78:0/798075198 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f13f8155580 msgr2=0x7f13f80c4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:19.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.410+0000 7f13fcff4700 1 --2- 172.21.15.78:0/798075198 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f13f8155580 0x7f13f80c4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:19.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.410+0000 7f13fcff4700 1 -- 172.21.15.78:0/798075198 >> v2:172.21.15.195:3300/0 conn(0x7f13f806b420 msgr2=0x7f13f80c9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:19.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.410+0000 7f13fcff4700 1 --2- 172.21.15.78:0/798075198 >> v2:172.21.15.195:3300/0 conn(0x7f13f806b420 0x7f13f80c9b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:19.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.410+0000 7f13fcff4700 1 -- 172.21.15.78:0/798075198 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f80c9040 con 0x7f13f814b5b0 2023-10-14T11:58:19.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.410+0000 7f13fd7f5700 1 --2- 172.21.15.78:0/798075198 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f13f8155580 0x7f13f80c4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:19.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.410+0000 7f13fcff4700 1 --2- 172.21.15.78:0/798075198 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13f814b5b0 0x7f13f80c8240 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f13f808c4e0 tx=0x7f13ec009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a3d6e31e527c9cc server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:19.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.411+0000 7f13f77fe700 1 -- 172.21.15.78:0/798075198 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ec00b870 con 0x7f13f814b5b0 2023-10-14T11:58:19.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.411+0000 7f13f77fe700 1 -- 172.21.15.78:0/798075198 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13ec00b9d0 con 0x7f13f814b5b0 2023-10-14T11:58:19.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.411+0000 7f13f77fe700 1 -- 172.21.15.78:0/798075198 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ec00bd60 con 0x7f13f814b5b0 2023-10-14T11:58:19.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.411+0000 7f13ff258700 1 -- 172.21.15.78:0/798075198 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13f814b5b0 msgr2=0x7f13f80c8240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:19.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.411+0000 7f13ff258700 1 --2- 172.21.15.78:0/798075198 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13f814b5b0 0x7f13f80c8240 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f13f808c4e0 tx=0x7f13ec009bd0 comp rx=0 tx=0).stop 2023-10-14T11:58:19.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.411+0000 7f13ff258700 1 -- 172.21.15.78:0/798075198 shutdown_connections 2023-10-14T11:58:19.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.411+0000 7f13ff258700 1 --2- 172.21.15.78:0/798075198 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f13f8155580 0x7f13f80c4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:19.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.411+0000 7f13ff258700 1 --2- 172.21.15.78:0/798075198 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13f814b5b0 0x7f13f80c8240 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:19.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.411+0000 7f13ff258700 1 --2- 172.21.15.78:0/798075198 >> v2:172.21.15.195:3300/0 conn(0x7f13f806b420 0x7f13f80c9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:19.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.411+0000 7f13ff258700 1 -- 172.21.15.78:0/798075198 >> 172.21.15.78:0/798075198 conn(0x7f13f80abe30 msgr2=0x7f13f80ac240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:19.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.411+0000 7f13ff258700 1 -- 172.21.15.78:0/798075198 shutdown_connections 2023-10-14T11:58:19.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.411+0000 7f13ff258700 1 -- 172.21.15.78:0/798075198 wait complete. 2023-10-14T11:58:19.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.412+0000 7f13ff258700 1 Processor -- start 2023-10-14T11:58:19.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.412+0000 7f13ff258700 1 -- start start 2023-10-14T11:58:19.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.413+0000 7f13ff258700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13f806b420 0x7f13f80c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:19.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.413+0000 7f13ff258700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f13f814b5b0 0x7f13f81dbc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:19.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.413+0000 7f13ff258700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f13f8155580 0x7f13f81dc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:19.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.413+0000 7f13ff258700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f81dc6c0 con 0x7f13f806b420 2023-10-14T11:58:19.424 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.413+0000 7f13ff258700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f80ae3e0 con 0x7f13f8155580 2023-10-14T11:58:19.424 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.413+0000 7f13ff258700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13f8149eb0 con 0x7f13f814b5b0 2023-10-14T11:58:19.424 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.413+0000 7f13fcff4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13f806b420 0x7f13f80c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:19.424 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.413+0000 7f13fcff4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13f806b420 0x7f13f80c1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44508/0 (socket says 172.21.15.78:44508) 2023-10-14T11:58:19.424 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.413+0000 7f13fcff4700 1 -- 172.21.15.78:0/2786973679 learned_addr learned my addr 172.21.15.78:0/2786973679 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:19.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.413+0000 7f13f7fff700 1 --2- 172.21.15.78:0/2786973679 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f13f814b5b0 0x7f13f81dbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:19.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.413+0000 7f13fd7f5700 1 --2- 172.21.15.78:0/2786973679 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f13f8155580 0x7f13f81dc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:19.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.414+0000 7f13fcff4700 1 -- 172.21.15.78:0/2786973679 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f13f814b5b0 msgr2=0x7f13f81dbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:19.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.414+0000 7f13fcff4700 1 --2- 172.21.15.78:0/2786973679 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f13f814b5b0 0x7f13f81dbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:19.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.414+0000 7f13fcff4700 1 -- 172.21.15.78:0/2786973679 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f13f8155580 msgr2=0x7f13f81dc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:19.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.414+0000 7f13fcff4700 1 --2- 172.21.15.78:0/2786973679 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f13f8155580 0x7f13f81dc180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:19.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.414+0000 7f13fcff4700 1 -- 172.21.15.78:0/2786973679 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ec009550 con 0x7f13f806b420 2023-10-14T11:58:19.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.414+0000 7f13fcff4700 1 --2- 172.21.15.78:0/2786973679 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13f806b420 0x7f13f80c1a50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f13ec000c00 tx=0x7f13ec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:19.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.414+0000 7f13f57fa700 1 -- 172.21.15.78:0/2786973679 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ec004480 con 0x7f13f806b420 2023-10-14T11:58:19.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.414+0000 7f13f57fa700 1 -- 172.21.15.78:0/2786973679 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13ec0045e0 con 0x7f13f806b420 2023-10-14T11:58:19.427 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.414+0000 7f13ff258700 1 -- 172.21.15.78:0/2786973679 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13f81ea010 con 0x7f13f806b420 2023-10-14T11:58:19.427 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.414+0000 7f13f57fa700 1 -- 172.21.15.78:0/2786973679 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ec0038e0 con 0x7f13f806b420 2023-10-14T11:58:19.427 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.414+0000 7f13ff258700 1 -- 172.21.15.78:0/2786973679 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13f81db160 con 0x7f13f806b420 2023-10-14T11:58:19.427 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.415+0000 7f13ff258700 1 -- 172.21.15.78:0/2786973679 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13f8074b80 con 0x7f13f806b420 2023-10-14T11:58:19.427 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.415+0000 7f13f57fa700 1 -- 172.21.15.78:0/2786973679 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f13ec021070 con 0x7f13f806b420 2023-10-14T11:58:19.428 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.416+0000 7f13f57fa700 1 --2- 172.21.15.78:0/2786973679 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f13dc06af00 0x7f13dc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:19.428 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.416+0000 7f13f57fa700 1 -- 172.21.15.78:0/2786973679 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f13ec08ab10 con 0x7f13f806b420 2023-10-14T11:58:19.428 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.416+0000 7f13f7fff700 1 --2- 172.21.15.78:0/2786973679 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f13dc06af00 0x7f13dc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:19.428 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.416+0000 7f13f7fff700 1 --2- 172.21.15.78:0/2786973679 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f13dc06af00 0x7f13dc06d3c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f13f81e7b00 tx=0x7f13e4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:19.428 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.420+0000 7f13f57fa700 1 -- 172.21.15.78:0/2786973679 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f13ec055c50 con 0x7f13f806b420 2023-10-14T11:58:19.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.609+0000 7f13ff258700 1 -- 172.21.15.78:0/2786973679 --> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f13f80c9600 con 0x7f13dc06af00 2023-10-14T11:58:19.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.616+0000 7f13f57fa700 1 -- 172.21.15.78:0/2786973679 <== mgr.4104 v2:172.21.15.78:6832/3171890074 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264833 (secure 0 0 0) 0x7f13e4009e40 con 0x7f13dc06af00 2023-10-14T11:58:19.621 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:19.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.618+0000 7f13ff258700 1 -- 172.21.15.78:0/2786973679 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f13dc06af00 msgr2=0x7f13dc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:19.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.618+0000 7f13ff258700 1 --2- 172.21.15.78:0/2786973679 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f13dc06af00 0x7f13dc06d3c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f13f81e7b00 tx=0x7f13e4008040 comp rx=0 tx=0).stop 2023-10-14T11:58:19.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.619+0000 7f13ff258700 1 -- 172.21.15.78:0/2786973679 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13f806b420 msgr2=0x7f13f80c1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:19.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.619+0000 7f13ff258700 1 --2- 172.21.15.78:0/2786973679 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13f806b420 0x7f13f80c1a50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f13ec000c00 tx=0x7f13ec004750 comp rx=0 tx=0).stop 2023-10-14T11:58:19.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.619+0000 7f13ff258700 1 -- 172.21.15.78:0/2786973679 shutdown_connections 2023-10-14T11:58:19.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.619+0000 7f13ff258700 1 --2- 172.21.15.78:0/2786973679 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f13dc06af00 0x7f13dc06d3c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:19.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.619+0000 7f13ff258700 1 --2- 172.21.15.78:0/2786973679 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f13f8155580 0x7f13f81dc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:19.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.619+0000 7f13ff258700 1 --2- 172.21.15.78:0/2786973679 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13f806b420 0x7f13f80c1a50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:19.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.619+0000 7f13ff258700 1 --2- 172.21.15.78:0/2786973679 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f13f814b5b0 0x7f13f81dbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:19.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.619+0000 7f13ff258700 1 -- 172.21.15.78:0/2786973679 >> 172.21.15.78:0/2786973679 conn(0x7f13f80abe30 msgr2=0x7f13f8152e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:19.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.619+0000 7f13ff258700 1 -- 172.21.15.78:0/2786973679 shutdown_connections 2023-10-14T11:58:19.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.619+0000 7f13ff258700 1 -- 172.21.15.78:0/2786973679 wait complete. 2023-10-14T11:58:19.625 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2023-10-14T11:58:19.632 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2023-10-14T11:58:18.958920+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218064,"kb_used_data":1876,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756656,"statfs":{"total":773094113280,"available":772870815744,"internally_reserved":0,"allocated":1921024,"data_stored":1783808,"data_compressed":65244,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.831539"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297796+0000","last_change":"2023-10-14T11:58:15.161449+0000","last_active":"2023-10-14T11:58:16.297796+0000","last_peered":"2023-10-14T11:58:16.297796+0000","last_clean":"2023-10-14T11:58:16.297796+0000","last_became_active":"2023-10-14T11:58:15.161100+0000","last_became_peered":"2023-10-14T11:58:15.161100+0000","last_unstale":"2023-10-14T11:58:16.297796+0000","last_undegraded":"2023-10-14T11:58:16.297796+0000","last_fullsized":"2023-10-14T11:58:16.297796+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:47:58.533238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.145107+0000","last_change":"2023-10-14T11:58:15.158210+0000","last_active":"2023-10-14T11:58:16.145107+0000","last_peered":"2023-10-14T11:58:16.145107+0000","last_clean":"2023-10-14T11:58:16.145107+0000","last_became_active":"2023-10-14T11:58:15.157886+0000","last_became_peered":"2023-10-14T11:58:15.157886+0000","last_unstale":"2023-10-14T11:58:16.145107+0000","last_undegraded":"2023-10-14T11:58:16.145107+0000","last_fullsized":"2023-10-14T11:58:16.145107+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:23:17.281352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296226+0000","last_change":"2023-10-14T11:58:15.161880+0000","last_active":"2023-10-14T11:58:16.296226+0000","last_peered":"2023-10-14T11:58:16.296226+0000","last_clean":"2023-10-14T11:58:16.296226+0000","last_became_active":"2023-10-14T11:58:15.161603+0000","last_became_peered":"2023-10-14T11:58:15.161603+0000","last_unstale":"2023-10-14T11:58:16.296226+0000","last_undegraded":"2023-10-14T11:58:16.296226+0000","last_fullsized":"2023-10-14T11:58:16.296226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:01:03.783116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296854+0000","last_change":"2023-10-14T11:58:15.167778+0000","last_active":"2023-10-14T11:58:16.296854+0000","last_peered":"2023-10-14T11:58:16.296854+0000","last_clean":"2023-10-14T11:58:16.296854+0000","last_became_active":"2023-10-14T11:58:15.167471+0000","last_became_peered":"2023-10-14T11:58:15.167471+0000","last_unstale":"2023-10-14T11:58:16.296854+0000","last_undegraded":"2023-10-14T11:58:16.296854+0000","last_fullsized":"2023-10-14T11:58:16.296854+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:07:58.884850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296653+0000","last_change":"2023-10-14T11:58:15.160788+0000","last_active":"2023-10-14T11:58:16.296653+0000","last_peered":"2023-10-14T11:58:16.296653+0000","last_clean":"2023-10-14T11:58:16.296653+0000","last_became_active":"2023-10-14T11:58:15.160525+0000","last_became_peered":"2023-10-14T11:58:15.160525+0000","last_unstale":"2023-10-14T11:58:16.296653+0000","last_undegraded":"2023-10-14T11:58:16.296653+0000","last_fullsized":"2023-10-14T11:58:16.296653+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:35:58.025887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297023+0000","last_change":"2023-10-14T11:58:15.167657+0000","last_active":"2023-10-14T11:58:16.297023+0000","last_peered":"2023-10-14T11:58:16.297023+0000","last_clean":"2023-10-14T11:58:16.297023+0000","last_became_active":"2023-10-14T11:58:15.167383+0000","last_became_peered":"2023-10-14T11:58:15.167383+0000","last_unstale":"2023-10-14T11:58:16.297023+0000","last_undegraded":"2023-10-14T11:58:16.297023+0000","last_fullsized":"2023-10-14T11:58:16.297023+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:53:19.926796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296834+0000","last_change":"2023-10-14T11:58:15.156244+0000","last_active":"2023-10-14T11:58:16.296834+0000","last_peered":"2023-10-14T11:58:16.296834+0000","last_clean":"2023-10-14T11:58:16.296834+0000","last_became_active":"2023-10-14T11:58:15.155621+0000","last_became_peered":"2023-10-14T11:58:15.155621+0000","last_unstale":"2023-10-14T11:58:16.296834+0000","last_undegraded":"2023-10-14T11:58:16.296834+0000","last_fullsized":"2023-10-14T11:58:16.296834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:07:58.884850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298542+0000","last_change":"2023-10-14T11:58:15.165865+0000","last_active":"2023-10-14T11:58:16.298542+0000","last_peered":"2023-10-14T11:58:16.298542+0000","last_clean":"2023-10-14T11:58:16.298542+0000","last_became_active":"2023-10-14T11:58:15.165613+0000","last_became_peered":"2023-10-14T11:58:15.165613+0000","last_unstale":"2023-10-14T11:58:16.298542+0000","last_undegraded":"2023-10-14T11:58:16.298542+0000","last_fullsized":"2023-10-14T11:58:16.298542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:32:38.145488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297728+0000","last_change":"2023-10-14T11:58:15.167047+0000","last_active":"2023-10-14T11:58:16.297728+0000","last_peered":"2023-10-14T11:58:16.297728+0000","last_clean":"2023-10-14T11:58:16.297728+0000","last_became_active":"2023-10-14T11:58:15.166362+0000","last_became_peered":"2023-10-14T11:58:15.166362+0000","last_unstale":"2023-10-14T11:58:16.297728+0000","last_undegraded":"2023-10-14T11:58:16.297728+0000","last_fullsized":"2023-10-14T11:58:16.297728+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:39:27.703491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144707+0000","last_change":"2023-10-14T11:58:15.164412+0000","last_active":"2023-10-14T11:58:16.144707+0000","last_peered":"2023-10-14T11:58:16.144707+0000","last_clean":"2023-10-14T11:58:16.144707+0000","last_became_active":"2023-10-14T11:58:15.164187+0000","last_became_peered":"2023-10-14T11:58:15.164187+0000","last_unstale":"2023-10-14T11:58:16.144707+0000","last_undegraded":"2023-10-14T11:58:16.144707+0000","last_fullsized":"2023-10-14T11:58:16.144707+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:59:03.464936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144466+0000","last_change":"2023-10-14T11:58:15.163857+0000","last_active":"2023-10-14T11:58:16.144466+0000","last_peered":"2023-10-14T11:58:16.144466+0000","last_clean":"2023-10-14T11:58:16.144466+0000","last_became_active":"2023-10-14T11:58:15.163575+0000","last_became_peered":"2023-10-14T11:58:15.163575+0000","last_unstale":"2023-10-14T11:58:16.144466+0000","last_undegraded":"2023-10-14T11:58:16.144466+0000","last_fullsized":"2023-10-14T11:58:16.144466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:01:03.783116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144222+0000","last_change":"2023-10-14T11:58:15.149569+0000","last_active":"2023-10-14T11:58:16.144222+0000","last_peered":"2023-10-14T11:58:16.144222+0000","last_clean":"2023-10-14T11:58:16.144222+0000","last_became_active":"2023-10-14T11:58:15.149161+0000","last_became_peered":"2023-10-14T11:58:15.149161+0000","last_unstale":"2023-10-14T11:58:16.144222+0000","last_undegraded":"2023-10-14T11:58:16.144222+0000","last_fullsized":"2023-10-14T11:58:16.144222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:19:30.143095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.299094+0000","last_change":"2023-10-14T11:58:15.147173+0000","last_active":"2023-10-14T11:58:16.299094+0000","last_peered":"2023-10-14T11:58:16.299094+0000","last_clean":"2023-10-14T11:58:16.299094+0000","last_became_active":"2023-10-14T11:58:15.146910+0000","last_became_peered":"2023-10-14T11:58:15.146910+0000","last_unstale":"2023-10-14T11:58:16.299094+0000","last_undegraded":"2023-10-14T11:58:16.299094+0000","last_fullsized":"2023-10-14T11:58:16.299094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:59:03.464936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296040+0000","last_change":"2023-10-14T11:58:15.169906+0000","last_active":"2023-10-14T11:58:16.296040+0000","last_peered":"2023-10-14T11:58:16.296040+0000","last_clean":"2023-10-14T11:58:16.296040+0000","last_became_active":"2023-10-14T11:58:15.169660+0000","last_became_peered":"2023-10-14T11:58:15.169660+0000","last_unstale":"2023-10-14T11:58:16.296040+0000","last_undegraded":"2023-10-14T11:58:16.296040+0000","last_fullsized":"2023-10-14T11:58:16.296040+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:32:38.145488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144820+0000","last_change":"2023-10-14T11:58:15.159353+0000","last_active":"2023-10-14T11:58:16.144820+0000","last_peered":"2023-10-14T11:58:16.144820+0000","last_clean":"2023-10-14T11:58:16.144820+0000","last_became_active":"2023-10-14T11:58:15.159095+0000","last_became_peered":"2023-10-14T11:58:15.159095+0000","last_unstale":"2023-10-14T11:58:16.144820+0000","last_undegraded":"2023-10-14T11:58:16.144820+0000","last_fullsized":"2023-10-14T11:58:16.144820+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:33:00.861183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296349+0000","last_change":"2023-10-14T11:58:15.169061+0000","last_active":"2023-10-14T11:58:16.296349+0000","last_peered":"2023-10-14T11:58:16.296349+0000","last_clean":"2023-10-14T11:58:16.296349+0000","last_became_active":"2023-10-14T11:58:15.168627+0000","last_became_peered":"2023-10-14T11:58:15.168627+0000","last_unstale":"2023-10-14T11:58:16.296349+0000","last_undegraded":"2023-10-14T11:58:16.296349+0000","last_fullsized":"2023-10-14T11:58:16.296349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:23:17.281352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297017+0000","last_change":"2023-10-14T11:58:15.162773+0000","last_active":"2023-10-14T11:58:16.297017+0000","last_peered":"2023-10-14T11:58:16.297017+0000","last_clean":"2023-10-14T11:58:16.297017+0000","last_became_active":"2023-10-14T11:58:15.162514+0000","last_became_peered":"2023-10-14T11:58:15.162514+0000","last_unstale":"2023-10-14T11:58:16.297017+0000","last_undegraded":"2023-10-14T11:58:16.297017+0000","last_fullsized":"2023-10-14T11:58:16.297017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:02:52.001714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298805+0000","last_change":"2023-10-14T11:58:15.164541+0000","last_active":"2023-10-14T11:58:16.298805+0000","last_peered":"2023-10-14T11:58:16.298805+0000","last_clean":"2023-10-14T11:58:16.298805+0000","last_became_active":"2023-10-14T11:58:15.164309+0000","last_became_peered":"2023-10-14T11:58:15.164309+0000","last_unstale":"2023-10-14T11:58:16.298805+0000","last_undegraded":"2023-10-14T11:58:16.298805+0000","last_fullsized":"2023-10-14T11:58:16.298805+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:22:04.803877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296451+0000","last_change":"2023-10-14T11:58:15.162171+0000","last_active":"2023-10-14T11:58:16.296451+0000","last_peered":"2023-10-14T11:58:16.296451+0000","last_clean":"2023-10-14T11:58:16.296451+0000","last_became_active":"2023-10-14T11:58:15.161894+0000","last_became_peered":"2023-10-14T11:58:15.161894+0000","last_unstale":"2023-10-14T11:58:16.296451+0000","last_undegraded":"2023-10-14T11:58:16.296451+0000","last_fullsized":"2023-10-14T11:58:16.296451+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:15:07.231894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296298+0000","last_change":"2023-10-14T11:58:14.145266+0000","last_active":"2023-10-14T11:58:16.296298+0000","last_peered":"2023-10-14T11:58:16.296298+0000","last_clean":"2023-10-14T11:58:16.296298+0000","last_became_active":"2023-10-14T11:58:14.144885+0000","last_became_peered":"2023-10-14T11:58:14.144885+0000","last_unstale":"2023-10-14T11:58:16.296298+0000","last_undegraded":"2023-10-14T11:58:16.296298+0000","last_fullsized":"2023-10-14T11:58:16.296298+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:06:27.985367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.299066+0000","last_change":"2023-10-14T11:58:15.148017+0000","last_active":"2023-10-14T11:58:16.299066+0000","last_peered":"2023-10-14T11:58:16.299066+0000","last_clean":"2023-10-14T11:58:16.299066+0000","last_became_active":"2023-10-14T11:58:15.147796+0000","last_became_peered":"2023-10-14T11:58:15.147796+0000","last_unstale":"2023-10-14T11:58:16.299066+0000","last_undegraded":"2023-10-14T11:58:16.299066+0000","last_fullsized":"2023-10-14T11:58:16.299066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:23:17.281352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.299129+0000","last_change":"2023-10-14T11:58:14.152335+0000","last_active":"2023-10-14T11:58:16.299129+0000","last_peered":"2023-10-14T11:58:16.299129+0000","last_clean":"2023-10-14T11:58:16.299129+0000","last_became_active":"2023-10-14T11:58:14.152064+0000","last_became_peered":"2023-10-14T11:58:14.152064+0000","last_unstale":"2023-10-14T11:58:16.299129+0000","last_undegraded":"2023-10-14T11:58:16.299129+0000","last_fullsized":"2023-10-14T11:58:16.299129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:21:22.993387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296225+0000","last_change":"2023-10-14T11:58:15.169510+0000","last_active":"2023-10-14T11:58:16.296225+0000","last_peered":"2023-10-14T11:58:16.296225+0000","last_clean":"2023-10-14T11:58:16.296225+0000","last_became_active":"2023-10-14T11:58:15.166488+0000","last_became_peered":"2023-10-14T11:58:15.166488+0000","last_unstale":"2023-10-14T11:58:16.296225+0000","last_undegraded":"2023-10-14T11:58:16.296225+0000","last_fullsized":"2023-10-14T11:58:16.296225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:23:17.281352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297447+0000","last_change":"2023-10-14T11:58:14.150333+0000","last_active":"2023-10-14T11:58:16.297447+0000","last_peered":"2023-10-14T11:58:16.297447+0000","last_clean":"2023-10-14T11:58:16.297447+0000","last_became_active":"2023-10-14T11:58:14.149979+0000","last_became_peered":"2023-10-14T11:58:14.149979+0000","last_unstale":"2023-10-14T11:58:16.297447+0000","last_undegraded":"2023-10-14T11:58:16.297447+0000","last_fullsized":"2023-10-14T11:58:16.297447+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:23:16.277717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297176+0000","last_change":"2023-10-14T11:58:15.163307+0000","last_active":"2023-10-14T11:58:16.297176+0000","last_peered":"2023-10-14T11:58:16.297176+0000","last_clean":"2023-10-14T11:58:16.297176+0000","last_became_active":"2023-10-14T11:58:15.163015+0000","last_became_peered":"2023-10-14T11:58:15.163015+0000","last_unstale":"2023-10-14T11:58:16.297176+0000","last_undegraded":"2023-10-14T11:58:16.297176+0000","last_fullsized":"2023-10-14T11:58:16.297176+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:21:10.166149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296924+0000","last_change":"2023-10-14T11:58:14.150931+0000","last_active":"2023-10-14T11:58:16.296924+0000","last_peered":"2023-10-14T11:58:16.296924+0000","last_clean":"2023-10-14T11:58:16.296924+0000","last_became_active":"2023-10-14T11:58:14.150629+0000","last_became_peered":"2023-10-14T11:58:14.150629+0000","last_unstale":"2023-10-14T11:58:16.296924+0000","last_undegraded":"2023-10-14T11:58:16.296924+0000","last_fullsized":"2023-10-14T11:58:16.296924+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:22:03.800242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297270+0000","last_change":"2023-10-14T11:58:15.163629+0000","last_active":"2023-10-14T11:58:16.297270+0000","last_peered":"2023-10-14T11:58:16.297270+0000","last_clean":"2023-10-14T11:58:16.297270+0000","last_became_active":"2023-10-14T11:58:15.163043+0000","last_became_peered":"2023-10-14T11:58:15.163043+0000","last_unstale":"2023-10-14T11:58:16.297270+0000","last_undegraded":"2023-10-14T11:58:16.297270+0000","last_fullsized":"2023-10-14T11:58:16.297270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:39:27.703491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298703+0000","last_change":"2023-10-14T11:58:14.154507+0000","last_active":"2023-10-14T11:58:16.298703+0000","last_peered":"2023-10-14T11:58:16.298703+0000","last_clean":"2023-10-14T11:58:16.298703+0000","last_became_active":"2023-10-14T11:58:14.154202+0000","last_became_peered":"2023-10-14T11:58:14.154202+0000","last_unstale":"2023-10-14T11:58:16.298703+0000","last_undegraded":"2023-10-14T11:58:16.298703+0000","last_fullsized":"2023-10-14T11:58:16.298703+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:59:45.791176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.143513+0000","last_change":"2023-10-14T11:58:15.158639+0000","last_active":"2023-10-14T11:58:16.143513+0000","last_peered":"2023-10-14T11:58:16.143513+0000","last_clean":"2023-10-14T11:58:16.143513+0000","last_became_active":"2023-10-14T11:58:15.158402+0000","last_became_peered":"2023-10-14T11:58:15.158402+0000","last_unstale":"2023-10-14T11:58:16.143513+0000","last_undegraded":"2023-10-14T11:58:16.143513+0000","last_fullsized":"2023-10-14T11:58:16.143513+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:21:23.997022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144263+0000","last_change":"2023-10-14T11:58:14.139502+0000","last_active":"2023-10-14T11:58:16.144263+0000","last_peered":"2023-10-14T11:58:16.144263+0000","last_clean":"2023-10-14T11:58:16.144263+0000","last_became_active":"2023-10-14T11:58:14.139053+0000","last_became_peered":"2023-10-14T11:58:14.139053+0000","last_unstale":"2023-10-14T11:58:16.144263+0000","last_undegraded":"2023-10-14T11:58:16.144263+0000","last_fullsized":"2023-10-14T11:58:16.144263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:10:10.291507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296162+0000","last_change":"2023-10-14T11:58:15.169858+0000","last_active":"2023-10-14T11:58:16.296162+0000","last_peered":"2023-10-14T11:58:16.296162+0000","last_clean":"2023-10-14T11:58:16.296162+0000","last_became_active":"2023-10-14T11:58:15.166724+0000","last_became_peered":"2023-10-14T11:58:15.166724+0000","last_unstale":"2023-10-14T11:58:16.296162+0000","last_undegraded":"2023-10-14T11:58:16.296162+0000","last_fullsized":"2023-10-14T11:58:16.296162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:45:35.707554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.299064+0000","last_change":"2023-10-14T11:58:14.154253+0000","last_active":"2023-10-14T11:58:16.299064+0000","last_peered":"2023-10-14T11:58:16.299064+0000","last_clean":"2023-10-14T11:58:16.299064+0000","last_became_active":"2023-10-14T11:58:14.152400+0000","last_became_peered":"2023-10-14T11:58:14.152400+0000","last_unstale":"2023-10-14T11:58:16.299064+0000","last_undegraded":"2023-10-14T11:58:16.299064+0000","last_fullsized":"2023-10-14T11:58:16.299064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:03:15.889637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298935+0000","last_change":"2023-10-14T11:58:15.167532+0000","last_active":"2023-10-14T11:58:16.298935+0000","last_peered":"2023-10-14T11:58:16.298935+0000","last_clean":"2023-10-14T11:58:16.298935+0000","last_became_active":"2023-10-14T11:58:15.167270+0000","last_became_peered":"2023-10-14T11:58:15.167270+0000","last_unstale":"2023-10-14T11:58:16.298935+0000","last_undegraded":"2023-10-14T11:58:16.298935+0000","last_fullsized":"2023-10-14T11:58:16.298935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:33:00.861183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296354+0000","last_change":"2023-10-14T11:58:14.147092+0000","last_active":"2023-10-14T11:58:16.296354+0000","last_peered":"2023-10-14T11:58:16.296354+0000","last_clean":"2023-10-14T11:58:16.296354+0000","last_became_active":"2023-10-14T11:58:14.146377+0000","last_became_peered":"2023-10-14T11:58:14.146377+0000","last_unstale":"2023-10-14T11:58:16.296354+0000","last_undegraded":"2023-10-14T11:58:16.296354+0000","last_fullsized":"2023-10-14T11:58:16.296354+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:25:37.999389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144924+0000","last_change":"2023-10-14T11:58:15.160286+0000","last_active":"2023-10-14T11:58:16.144924+0000","last_peered":"2023-10-14T11:58:16.144924+0000","last_clean":"2023-10-14T11:58:16.144924+0000","last_became_active":"2023-10-14T11:58:15.159308+0000","last_became_peered":"2023-10-14T11:58:15.159308+0000","last_unstale":"2023-10-14T11:58:16.144924+0000","last_undegraded":"2023-10-14T11:58:16.144924+0000","last_fullsized":"2023-10-14T11:58:16.144924+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:45:35.707554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144601+0000","last_change":"2023-10-14T11:58:14.148448+0000","last_active":"2023-10-14T11:58:16.144601+0000","last_peered":"2023-10-14T11:58:16.144601+0000","last_clean":"2023-10-14T11:58:16.144601+0000","last_became_active":"2023-10-14T11:58:14.148155+0000","last_became_peered":"2023-10-14T11:58:14.148155+0000","last_unstale":"2023-10-14T11:58:16.144601+0000","last_undegraded":"2023-10-14T11:58:16.144601+0000","last_fullsized":"2023-10-14T11:58:16.144601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:19:29.139460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296785+0000","last_change":"2023-10-14T11:58:15.166089+0000","last_active":"2023-10-14T11:58:16.296785+0000","last_peered":"2023-10-14T11:58:16.296785+0000","last_clean":"2023-10-14T11:58:16.296785+0000","last_became_active":"2023-10-14T11:58:15.164889+0000","last_became_peered":"2023-10-14T11:58:15.164889+0000","last_unstale":"2023-10-14T11:58:16.296785+0000","last_undegraded":"2023-10-14T11:58:16.296785+0000","last_fullsized":"2023-10-14T11:58:16.296785+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:15.482221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296897+0000","last_change":"2023-10-14T11:58:14.150455+0000","last_active":"2023-10-14T11:58:16.296897+0000","last_peered":"2023-10-14T11:58:16.296897+0000","last_clean":"2023-10-14T11:58:16.296897+0000","last_became_active":"2023-10-14T11:58:14.150195+0000","last_became_peered":"2023-10-14T11:58:14.150195+0000","last_unstale":"2023-10-14T11:58:16.296897+0000","last_undegraded":"2023-10-14T11:58:16.296897+0000","last_fullsized":"2023-10-14T11:58:16.296897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:32:59.857548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296229+0000","last_change":"2023-10-14T11:58:15.170459+0000","last_active":"2023-10-14T11:58:16.296229+0000","last_peered":"2023-10-14T11:58:16.296229+0000","last_clean":"2023-10-14T11:58:16.296229+0000","last_became_active":"2023-10-14T11:58:15.170096+0000","last_became_peered":"2023-10-14T11:58:15.170096+0000","last_unstale":"2023-10-14T11:58:16.296229+0000","last_undegraded":"2023-10-14T11:58:16.296229+0000","last_fullsized":"2023-10-14T11:58:16.296229+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:21:15.443882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.145168+0000","last_change":"2023-10-14T11:58:14.147259+0000","last_active":"2023-10-14T11:58:16.145168+0000","last_peered":"2023-10-14T11:58:16.145168+0000","last_clean":"2023-10-14T11:58:16.145168+0000","last_became_active":"2023-10-14T11:58:14.146982+0000","last_became_peered":"2023-10-14T11:58:14.146982+0000","last_unstale":"2023-10-14T11:58:16.145168+0000","last_undegraded":"2023-10-14T11:58:16.145168+0000","last_fullsized":"2023-10-14T11:58:16.145168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:22:03.800242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144175+0000","last_change":"2023-10-14T11:58:15.158917+0000","last_active":"2023-10-14T11:58:16.144175+0000","last_peered":"2023-10-14T11:58:16.144175+0000","last_clean":"2023-10-14T11:58:16.144175+0000","last_became_active":"2023-10-14T11:58:15.158578+0000","last_became_peered":"2023-10-14T11:58:15.158578+0000","last_unstale":"2023-10-14T11:58:16.144175+0000","last_undegraded":"2023-10-14T11:58:16.144175+0000","last_fullsized":"2023-10-14T11:58:16.144175+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:06:28.989002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144380+0000","last_change":"2023-10-14T11:58:14.152849+0000","last_active":"2023-10-14T11:58:16.144380+0000","last_peered":"2023-10-14T11:58:16.144380+0000","last_clean":"2023-10-14T11:58:16.144380+0000","last_became_active":"2023-10-14T11:58:14.152544+0000","last_became_peered":"2023-10-14T11:58:14.152544+0000","last_unstale":"2023-10-14T11:58:16.144380+0000","last_undegraded":"2023-10-14T11:58:16.144380+0000","last_fullsized":"2023-10-14T11:58:16.144380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:30:50.303297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297109+0000","last_change":"2023-10-14T11:58:15.160605+0000","last_active":"2023-10-14T11:58:16.297109+0000","last_peered":"2023-10-14T11:58:16.297109+0000","last_clean":"2023-10-14T11:58:16.297109+0000","last_became_active":"2023-10-14T11:58:15.160323+0000","last_became_peered":"2023-10-14T11:58:15.160323+0000","last_unstale":"2023-10-14T11:58:16.297109+0000","last_undegraded":"2023-10-14T11:58:16.297109+0000","last_fullsized":"2023-10-14T11:58:16.297109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:53:19.926796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296162+0000","last_change":"2023-10-14T11:58:14.145911+0000","last_active":"2023-10-14T11:58:16.296162+0000","last_peered":"2023-10-14T11:58:16.296162+0000","last_clean":"2023-10-14T11:58:16.296162+0000","last_became_active":"2023-10-14T11:58:14.144322+0000","last_became_peered":"2023-10-14T11:58:14.144322+0000","last_unstale":"2023-10-14T11:58:16.296162+0000","last_undegraded":"2023-10-14T11:58:16.296162+0000","last_fullsized":"2023-10-14T11:58:16.296162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:32:03.667630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.295965+0000","last_change":"2023-10-14T11:58:15.165189+0000","last_active":"2023-10-14T11:58:16.295965+0000","last_peered":"2023-10-14T11:58:16.295965+0000","last_clean":"2023-10-14T11:58:16.295965+0000","last_became_active":"2023-10-14T11:58:15.164911+0000","last_became_peered":"2023-10-14T11:58:15.164911+0000","last_unstale":"2023-10-14T11:58:16.295965+0000","last_undegraded":"2023-10-14T11:58:16.295965+0000","last_fullsized":"2023-10-14T11:58:16.295965+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:10:17.502466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297270+0000","last_change":"2023-10-14T11:58:14.152736+0000","last_active":"2023-10-14T11:58:16.297270+0000","last_peered":"2023-10-14T11:58:16.297270+0000","last_clean":"2023-10-14T11:58:16.297270+0000","last_became_active":"2023-10-14T11:58:14.152401+0000","last_became_peered":"2023-10-14T11:58:14.152401+0000","last_unstale":"2023-10-14T11:58:16.297270+0000","last_undegraded":"2023-10-14T11:58:16.297270+0000","last_fullsized":"2023-10-14T11:58:16.297270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:01:02.779481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296948+0000","last_change":"2023-10-14T11:58:15.155365+0000","last_active":"2023-10-14T11:58:16.296948+0000","last_peered":"2023-10-14T11:58:16.296948+0000","last_clean":"2023-10-14T11:58:16.296948+0000","last_became_active":"2023-10-14T11:58:15.155102+0000","last_became_peered":"2023-10-14T11:58:15.155102+0000","last_unstale":"2023-10-14T11:58:16.296948+0000","last_undegraded":"2023-10-14T11:58:16.296948+0000","last_fullsized":"2023-10-14T11:58:16.296948+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:02:52.001714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.143938+0000","last_change":"2023-10-14T11:58:14.148862+0000","last_active":"2023-10-14T11:58:16.143938+0000","last_peered":"2023-10-14T11:58:16.143938+0000","last_clean":"2023-10-14T11:58:16.143938+0000","last_became_active":"2023-10-14T11:58:14.148409+0000","last_became_peered":"2023-10-14T11:58:14.148409+0000","last_unstale":"2023-10-14T11:58:16.143938+0000","last_undegraded":"2023-10-14T11:58:16.143938+0000","last_fullsized":"2023-10-14T11:58:16.143938+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:06:27.985367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.143490+0000","last_change":"2023-10-14T11:58:15.156428+0000","last_active":"2023-10-14T11:58:16.143490+0000","last_peered":"2023-10-14T11:58:16.143490+0000","last_clean":"2023-10-14T11:58:16.143490+0000","last_became_active":"2023-10-14T11:58:15.155818+0000","last_became_peered":"2023-10-14T11:58:15.155818+0000","last_unstale":"2023-10-14T11:58:16.143490+0000","last_undegraded":"2023-10-14T11:58:16.143490+0000","last_fullsized":"2023-10-14T11:58:16.143490+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:33:00.861183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298391+0000","last_change":"2023-10-14T11:58:14.149643+0000","last_active":"2023-10-14T11:58:16.298391+0000","last_peered":"2023-10-14T11:58:16.298391+0000","last_clean":"2023-10-14T11:58:16.298391+0000","last_became_active":"2023-10-14T11:58:14.149330+0000","last_became_peered":"2023-10-14T11:58:14.149330+0000","last_unstale":"2023-10-14T11:58:16.298391+0000","last_undegraded":"2023-10-14T11:58:16.298391+0000","last_fullsized":"2023-10-14T11:58:16.298391+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:14:56.067093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296718+0000","last_change":"2023-10-14T11:58:15.156412+0000","last_active":"2023-10-14T11:58:16.296718+0000","last_peered":"2023-10-14T11:58:16.296718+0000","last_clean":"2023-10-14T11:58:16.296718+0000","last_became_active":"2023-10-14T11:58:15.155951+0000","last_became_peered":"2023-10-14T11:58:15.155951+0000","last_unstale":"2023-10-14T11:58:16.296718+0000","last_undegraded":"2023-10-14T11:58:16.296718+0000","last_fullsized":"2023-10-14T11:58:16.296718+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:15.482221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.143832+0000","last_change":"2023-10-14T11:58:14.135648+0000","last_active":"2023-10-14T11:58:16.143832+0000","last_peered":"2023-10-14T11:58:16.143832+0000","last_clean":"2023-10-14T11:58:16.143832+0000","last_became_active":"2023-10-14T11:58:14.135104+0000","last_became_peered":"2023-10-14T11:58:14.135104+0000","last_unstale":"2023-10-14T11:58:16.143832+0000","last_undegraded":"2023-10-14T11:58:16.143832+0000","last_fullsized":"2023-10-14T11:58:16.143832+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:47:41.184305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297156+0000","last_change":"2023-10-14T11:58:15.157354+0000","last_active":"2023-10-14T11:58:16.297156+0000","last_peered":"2023-10-14T11:58:16.297156+0000","last_clean":"2023-10-14T11:58:16.297156+0000","last_became_active":"2023-10-14T11:58:15.157084+0000","last_became_peered":"2023-10-14T11:58:15.157084+0000","last_unstale":"2023-10-14T11:58:16.297156+0000","last_undegraded":"2023-10-14T11:58:16.297156+0000","last_fullsized":"2023-10-14T11:58:16.297156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:32:40.379634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144935+0000","last_change":"2023-10-14T11:58:14.148246+0000","last_active":"2023-10-14T11:58:16.144935+0000","last_peered":"2023-10-14T11:58:16.144935+0000","last_clean":"2023-10-14T11:58:16.144935+0000","last_became_active":"2023-10-14T11:58:14.147943+0000","last_became_peered":"2023-10-14T11:58:14.147943+0000","last_unstale":"2023-10-14T11:58:16.144935+0000","last_undegraded":"2023-10-14T11:58:16.144935+0000","last_fullsized":"2023-10-14T11:58:16.144935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:19:29.139460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297125+0000","last_change":"2023-10-14T11:58:15.162066+0000","last_active":"2023-10-14T11:58:16.297125+0000","last_peered":"2023-10-14T11:58:16.297125+0000","last_clean":"2023-10-14T11:58:16.297125+0000","last_became_active":"2023-10-14T11:58:15.161798+0000","last_became_peered":"2023-10-14T11:58:15.161798+0000","last_unstale":"2023-10-14T11:58:16.297125+0000","last_undegraded":"2023-10-14T11:58:16.297125+0000","last_fullsized":"2023-10-14T11:58:16.297125+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:32:40.379634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297269+0000","last_change":"2023-10-14T11:58:14.147024+0000","last_active":"2023-10-14T11:58:16.297269+0000","last_peered":"2023-10-14T11:58:16.297269+0000","last_clean":"2023-10-14T11:58:16.297269+0000","last_became_active":"2023-10-14T11:58:14.146675+0000","last_became_peered":"2023-10-14T11:58:14.146675+0000","last_unstale":"2023-10-14T11:58:16.297269+0000","last_undegraded":"2023-10-14T11:58:16.297269+0000","last_fullsized":"2023-10-14T11:58:16.297269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:45:34.703919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.143832+0000","last_change":"2023-10-14T11:58:15.157978+0000","last_active":"2023-10-14T11:58:16.143832+0000","last_peered":"2023-10-14T11:58:16.143832+0000","last_clean":"2023-10-14T11:58:16.143832+0000","last_became_active":"2023-10-14T11:58:15.157698+0000","last_became_peered":"2023-10-14T11:58:15.157698+0000","last_unstale":"2023-10-14T11:58:16.143832+0000","last_undegraded":"2023-10-14T11:58:16.143832+0000","last_fullsized":"2023-10-14T11:58:16.143832+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:10:11.295142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144794+0000","last_change":"2023-10-14T11:58:15.164985+0000","last_active":"2023-10-14T11:58:16.144794+0000","last_peered":"2023-10-14T11:58:16.144794+0000","last_clean":"2023-10-14T11:58:16.144794+0000","last_became_active":"2023-10-14T11:58:15.164355+0000","last_became_peered":"2023-10-14T11:58:15.164355+0000","last_unstale":"2023-10-14T11:58:16.144794+0000","last_undegraded":"2023-10-14T11:58:16.144794+0000","last_fullsized":"2023-10-14T11:58:16.144794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:25:39.003024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298382+0000","last_change":"2023-10-14T11:58:14.153483+0000","last_active":"2023-10-14T11:58:16.298382+0000","last_peered":"2023-10-14T11:58:16.298382+0000","last_clean":"2023-10-14T11:58:16.298382+0000","last_became_active":"2023-10-14T11:58:14.153174+0000","last_became_peered":"2023-10-14T11:58:14.153174+0000","last_unstale":"2023-10-14T11:58:16.298382+0000","last_undegraded":"2023-10-14T11:58:16.298382+0000","last_fullsized":"2023-10-14T11:58:16.298382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:06:27.985367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298697+0000","last_change":"2023-10-14T11:58:15.171141+0000","last_active":"2023-10-14T11:58:16.298697+0000","last_peered":"2023-10-14T11:58:16.298697+0000","last_clean":"2023-10-14T11:58:16.298697+0000","last_became_active":"2023-10-14T11:58:15.170853+0000","last_became_peered":"2023-10-14T11:58:15.170853+0000","last_unstale":"2023-10-14T11:58:16.298697+0000","last_undegraded":"2023-10-14T11:58:16.298697+0000","last_fullsized":"2023-10-14T11:58:16.298697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:45:35.707554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296962+0000","last_change":"2023-10-14T11:58:14.153660+0000","last_active":"2023-10-14T11:58:16.296962+0000","last_peered":"2023-10-14T11:58:16.296962+0000","last_clean":"2023-10-14T11:58:16.296962+0000","last_became_active":"2023-10-14T11:58:14.153419+0000","last_became_peered":"2023-10-14T11:58:14.153419+0000","last_unstale":"2023-10-14T11:58:16.296962+0000","last_undegraded":"2023-10-14T11:58:16.296962+0000","last_fullsized":"2023-10-14T11:58:16.296962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:06:27.985367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T11:58:15.149401+0000","last_change":"2023-10-14T11:58:10.616726+0000","last_active":"2023-10-14T11:58:15.149401+0000","last_peered":"2023-10-14T11:58:15.149401+0000","last_clean":"2023-10-14T11:58:15.149401+0000","last_became_active":"2023-10-14T11:58:10.615628+0000","last_became_peered":"2023-10-14T11:58:10.615628+0000","last_unstale":"2023-10-14T11:58:15.149401+0000","last_undegraded":"2023-10-14T11:58:15.149401+0000","last_fullsized":"2023-10-14T11:58:15.149401+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:09.001265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:09.001265+0000","last_clean_scrub_stamp":"2023-10-14T11:58:09.001265+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:22:50.496050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.295927+0000","last_change":"2023-10-14T11:58:15.173632+0000","last_active":"2023-10-14T11:58:16.295927+0000","last_peered":"2023-10-14T11:58:16.295927+0000","last_clean":"2023-10-14T11:58:16.295927+0000","last_became_active":"2023-10-14T11:58:15.172996+0000","last_became_peered":"2023-10-14T11:58:15.172996+0000","last_unstale":"2023-10-14T11:58:16.295927+0000","last_undegraded":"2023-10-14T11:58:16.295927+0000","last_fullsized":"2023-10-14T11:58:16.295927+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:22:04.803877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296824+0000","last_change":"2023-10-14T11:58:14.150078+0000","last_active":"2023-10-14T11:58:16.296824+0000","last_peered":"2023-10-14T11:58:16.296824+0000","last_clean":"2023-10-14T11:58:16.296824+0000","last_became_active":"2023-10-14T11:58:14.149788+0000","last_became_peered":"2023-10-14T11:58:14.149788+0000","last_unstale":"2023-10-14T11:58:16.296824+0000","last_undegraded":"2023-10-14T11:58:16.296824+0000","last_fullsized":"2023-10-14T11:58:16.296824+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:10:10.291507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297353+0000","last_change":"2023-10-14T11:58:14.151386+0000","last_active":"2023-10-14T11:58:16.297353+0000","last_peered":"2023-10-14T11:58:16.297353+0000","last_clean":"2023-10-14T11:58:16.297353+0000","last_became_active":"2023-10-14T11:58:14.151041+0000","last_became_peered":"2023-10-14T11:58:14.151041+0000","last_unstale":"2023-10-14T11:58:16.297353+0000","last_undegraded":"2023-10-14T11:58:16.297353+0000","last_fullsized":"2023-10-14T11:58:16.297353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:21:14.440247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144942+0000","last_change":"2023-10-14T11:58:15.168958+0000","last_active":"2023-10-14T11:58:16.144942+0000","last_peered":"2023-10-14T11:58:16.144942+0000","last_clean":"2023-10-14T11:58:16.144942+0000","last_became_active":"2023-10-14T11:58:15.164516+0000","last_became_peered":"2023-10-14T11:58:15.164516+0000","last_unstale":"2023-10-14T11:58:16.144942+0000","last_undegraded":"2023-10-14T11:58:16.144942+0000","last_fullsized":"2023-10-14T11:58:16.144942+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:01:03.783116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144989+0000","last_change":"2023-10-14T11:58:14.148617+0000","last_active":"2023-10-14T11:58:16.144989+0000","last_peered":"2023-10-14T11:58:16.144989+0000","last_clean":"2023-10-14T11:58:16.144989+0000","last_became_active":"2023-10-14T11:58:14.148324+0000","last_became_peered":"2023-10-14T11:58:14.148324+0000","last_unstale":"2023-10-14T11:58:16.144989+0000","last_undegraded":"2023-10-14T11:58:16.144989+0000","last_fullsized":"2023-10-14T11:58:16.144989+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:10:16.498831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.299219+0000","last_change":"2023-10-14T11:58:15.162441+0000","last_active":"2023-10-14T11:58:16.299219+0000","last_peered":"2023-10-14T11:58:16.299219+0000","last_clean":"2023-10-14T11:58:16.299219+0000","last_became_active":"2023-10-14T11:58:15.162047+0000","last_became_peered":"2023-10-14T11:58:15.162047+0000","last_unstale":"2023-10-14T11:58:16.299219+0000","last_undegraded":"2023-10-14T11:58:16.299219+0000","last_fullsized":"2023-10-14T11:58:16.299219+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:15.482221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144522+0000","last_change":"2023-10-14T11:58:14.151993+0000","last_active":"2023-10-14T11:58:16.144522+0000","last_peered":"2023-10-14T11:58:16.144522+0000","last_clean":"2023-10-14T11:58:16.144522+0000","last_became_active":"2023-10-14T11:58:14.151721+0000","last_became_peered":"2023-10-14T11:58:14.151721+0000","last_unstale":"2023-10-14T11:58:16.144522+0000","last_undegraded":"2023-10-14T11:58:16.144522+0000","last_fullsized":"2023-10-14T11:58:16.144522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:21:22.993387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296156+0000","last_change":"2023-10-14T11:58:15.170037+0000","last_active":"2023-10-14T11:58:16.296156+0000","last_peered":"2023-10-14T11:58:16.296156+0000","last_clean":"2023-10-14T11:58:16.296156+0000","last_became_active":"2023-10-14T11:58:15.169745+0000","last_became_peered":"2023-10-14T11:58:15.169745+0000","last_unstale":"2023-10-14T11:58:16.296156+0000","last_undegraded":"2023-10-14T11:58:16.296156+0000","last_fullsized":"2023-10-14T11:58:16.296156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:33:00.861183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296261+0000","last_change":"2023-10-14T11:58:14.151181+0000","last_active":"2023-10-14T11:58:16.296261+0000","last_peered":"2023-10-14T11:58:16.296261+0000","last_clean":"2023-10-14T11:58:16.296261+0000","last_became_active":"2023-10-14T11:58:14.150859+0000","last_became_peered":"2023-10-14T11:58:14.150859+0000","last_unstale":"2023-10-14T11:58:16.296261+0000","last_undegraded":"2023-10-14T11:58:16.296261+0000","last_fullsized":"2023-10-14T11:58:16.296261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:21:22.993387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144028+0000","last_change":"2023-10-14T11:58:15.168925+0000","last_active":"2023-10-14T11:58:16.144028+0000","last_peered":"2023-10-14T11:58:16.144028+0000","last_clean":"2023-10-14T11:58:16.144028+0000","last_became_active":"2023-10-14T11:58:15.168534+0000","last_became_peered":"2023-10-14T11:58:15.168534+0000","last_unstale":"2023-10-14T11:58:16.144028+0000","last_undegraded":"2023-10-14T11:58:16.144028+0000","last_fullsized":"2023-10-14T11:58:16.144028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:58:44.511366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144781+0000","last_change":"2023-10-14T11:58:14.150372+0000","last_active":"2023-10-14T11:58:16.144781+0000","last_peered":"2023-10-14T11:58:16.144781+0000","last_clean":"2023-10-14T11:58:16.144781+0000","last_became_active":"2023-10-14T11:58:14.150095+0000","last_became_peered":"2023-10-14T11:58:14.150095+0000","last_unstale":"2023-10-14T11:58:16.144781+0000","last_undegraded":"2023-10-14T11:58:16.144781+0000","last_fullsized":"2023-10-14T11:58:16.144781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:32:37.141853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298587+0000","last_change":"2023-10-14T11:58:15.158813+0000","last_active":"2023-10-14T11:58:16.298587+0000","last_peered":"2023-10-14T11:58:16.298587+0000","last_clean":"2023-10-14T11:58:16.298587+0000","last_became_active":"2023-10-14T11:58:15.158536+0000","last_became_peered":"2023-10-14T11:58:15.158536+0000","last_unstale":"2023-10-14T11:58:16.298587+0000","last_undegraded":"2023-10-14T11:58:16.298587+0000","last_fullsized":"2023-10-14T11:58:16.298587+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:25:39.003024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.145036+0000","last_change":"2023-10-14T11:58:14.153044+0000","last_active":"2023-10-14T11:58:16.145036+0000","last_peered":"2023-10-14T11:58:16.145036+0000","last_clean":"2023-10-14T11:58:16.145036+0000","last_became_active":"2023-10-14T11:58:14.152746+0000","last_became_peered":"2023-10-14T11:58:14.152746+0000","last_unstale":"2023-10-14T11:58:16.145036+0000","last_undegraded":"2023-10-14T11:58:16.145036+0000","last_fullsized":"2023-10-14T11:58:16.145036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:30:50.303297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298842+0000","last_change":"2023-10-14T11:58:15.174434+0000","last_active":"2023-10-14T11:58:16.298842+0000","last_peered":"2023-10-14T11:58:16.298842+0000","last_clean":"2023-10-14T11:58:16.298842+0000","last_became_active":"2023-10-14T11:58:15.174141+0000","last_became_peered":"2023-10-14T11:58:15.174141+0000","last_unstale":"2023-10-14T11:58:16.298842+0000","last_undegraded":"2023-10-14T11:58:16.298842+0000","last_fullsized":"2023-10-14T11:58:16.298842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:21:15.443882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296762+0000","last_change":"2023-10-14T11:58:14.149244+0000","last_active":"2023-10-14T11:58:16.296762+0000","last_peered":"2023-10-14T11:58:16.296762+0000","last_clean":"2023-10-14T11:58:16.296762+0000","last_became_active":"2023-10-14T11:58:14.148960+0000","last_became_peered":"2023-10-14T11:58:14.148960+0000","last_unstale":"2023-10-14T11:58:16.296762+0000","last_undegraded":"2023-10-14T11:58:16.296762+0000","last_fullsized":"2023-10-14T11:58:16.296762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:32:03.667630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144154+0000","last_change":"2023-10-14T11:58:15.150214+0000","last_active":"2023-10-14T11:58:16.144154+0000","last_peered":"2023-10-14T11:58:16.144154+0000","last_clean":"2023-10-14T11:58:16.144154+0000","last_became_active":"2023-10-14T11:58:15.149927+0000","last_became_peered":"2023-10-14T11:58:15.149927+0000","last_unstale":"2023-10-14T11:58:16.144154+0000","last_undegraded":"2023-10-14T11:58:16.144154+0000","last_fullsized":"2023-10-14T11:58:16.144154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:45:35.707554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298684+0000","last_change":"2023-10-14T11:58:14.153886+0000","last_active":"2023-10-14T11:58:16.298684+0000","last_peered":"2023-10-14T11:58:16.298684+0000","last_clean":"2023-10-14T11:58:16.298684+0000","last_became_active":"2023-10-14T11:58:14.153365+0000","last_became_peered":"2023-10-14T11:58:14.153365+0000","last_unstale":"2023-10-14T11:58:16.298684+0000","last_undegraded":"2023-10-14T11:58:16.298684+0000","last_fullsized":"2023-10-14T11:58:16.298684+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:21:22.993387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297257+0000","last_change":"2023-10-14T11:58:15.168448+0000","last_active":"2023-10-14T11:58:16.297257+0000","last_peered":"2023-10-14T11:58:16.297257+0000","last_clean":"2023-10-14T11:58:16.297257+0000","last_became_active":"2023-10-14T11:58:15.167757+0000","last_became_peered":"2023-10-14T11:58:15.167757+0000","last_unstale":"2023-10-14T11:58:16.297257+0000","last_undegraded":"2023-10-14T11:58:16.297257+0000","last_fullsized":"2023-10-14T11:58:16.297257+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:21:10.166149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.145150+0000","last_change":"2023-10-14T11:58:14.145505+0000","last_active":"2023-10-14T11:58:16.145150+0000","last_peered":"2023-10-14T11:58:16.145150+0000","last_clean":"2023-10-14T11:58:16.145150+0000","last_became_active":"2023-10-14T11:58:14.145218+0000","last_became_peered":"2023-10-14T11:58:14.145218+0000","last_unstale":"2023-10-14T11:58:16.145150+0000","last_undegraded":"2023-10-14T11:58:16.145150+0000","last_fullsized":"2023-10-14T11:58:16.145150+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:32:03.667630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297202+0000","last_change":"2023-10-14T11:58:15.162290+0000","last_active":"2023-10-14T11:58:16.297202+0000","last_peered":"2023-10-14T11:58:16.297202+0000","last_clean":"2023-10-14T11:58:16.297202+0000","last_became_active":"2023-10-14T11:58:15.161987+0000","last_became_peered":"2023-10-14T11:58:15.161987+0000","last_unstale":"2023-10-14T11:58:16.297202+0000","last_undegraded":"2023-10-14T11:58:16.297202+0000","last_fullsized":"2023-10-14T11:58:16.297202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:42:47.990050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298895+0000","last_change":"2023-10-14T11:58:14.154431+0000","last_active":"2023-10-14T11:58:16.298895+0000","last_peered":"2023-10-14T11:58:16.298895+0000","last_clean":"2023-10-14T11:58:16.298895+0000","last_became_active":"2023-10-14T11:58:14.153698+0000","last_became_peered":"2023-10-14T11:58:14.153698+0000","last_unstale":"2023-10-14T11:58:16.298895+0000","last_undegraded":"2023-10-14T11:58:16.298895+0000","last_fullsized":"2023-10-14T11:58:16.298895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:10:10.291507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297698+0000","last_change":"2023-10-14T11:58:15.160591+0000","last_active":"2023-10-14T11:58:16.297698+0000","last_peered":"2023-10-14T11:58:16.297698+0000","last_clean":"2023-10-14T11:58:16.297698+0000","last_became_active":"2023-10-14T11:58:15.160280+0000","last_became_peered":"2023-10-14T11:58:15.160280+0000","last_unstale":"2023-10-14T11:58:16.297698+0000","last_undegraded":"2023-10-14T11:58:16.297698+0000","last_fullsized":"2023-10-14T11:58:16.297698+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:42:47.990050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.143334+0000","last_change":"2023-10-14T11:58:15.156633+0000","last_active":"2023-10-14T11:58:16.143334+0000","last_peered":"2023-10-14T11:58:16.143334+0000","last_clean":"2023-10-14T11:58:16.143334+0000","last_became_active":"2023-10-14T11:58:15.156176+0000","last_became_peered":"2023-10-14T11:58:15.156176+0000","last_unstale":"2023-10-14T11:58:16.143334+0000","last_undegraded":"2023-10-14T11:58:16.143334+0000","last_fullsized":"2023-10-14T11:58:16.143334+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:25:39.003024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298424+0000","last_change":"2023-10-14T11:58:15.165873+0000","last_active":"2023-10-14T11:58:16.298424+0000","last_peered":"2023-10-14T11:58:16.298424+0000","last_clean":"2023-10-14T11:58:16.298424+0000","last_became_active":"2023-10-14T11:58:15.165592+0000","last_became_peered":"2023-10-14T11:58:15.165592+0000","last_unstale":"2023-10-14T11:58:16.298424+0000","last_undegraded":"2023-10-14T11:58:16.298424+0000","last_fullsized":"2023-10-14T11:58:16.298424+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:32:04.671265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296749+0000","last_change":"2023-10-14T11:58:15.165467+0000","last_active":"2023-10-14T11:58:16.296749+0000","last_peered":"2023-10-14T11:58:16.296749+0000","last_clean":"2023-10-14T11:58:16.296749+0000","last_became_active":"2023-10-14T11:58:15.165032+0000","last_became_peered":"2023-10-14T11:58:15.165032+0000","last_unstale":"2023-10-14T11:58:16.296749+0000","last_undegraded":"2023-10-14T11:58:16.296749+0000","last_fullsized":"2023-10-14T11:58:16.296749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:35:58.025887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296067+0000","last_change":"2023-10-14T11:58:15.170357+0000","last_active":"2023-10-14T11:58:16.296067+0000","last_peered":"2023-10-14T11:58:16.296067+0000","last_clean":"2023-10-14T11:58:16.296067+0000","last_became_active":"2023-10-14T11:58:15.170062+0000","last_became_peered":"2023-10-14T11:58:15.170062+0000","last_unstale":"2023-10-14T11:58:16.296067+0000","last_undegraded":"2023-10-14T11:58:16.296067+0000","last_fullsized":"2023-10-14T11:58:16.296067+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:25:39.003024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.299070+0000","last_change":"2023-10-14T11:58:15.156230+0000","last_active":"2023-10-14T11:58:16.299070+0000","last_peered":"2023-10-14T11:58:16.299070+0000","last_clean":"2023-10-14T11:58:16.299070+0000","last_became_active":"2023-10-14T11:58:15.155970+0000","last_became_peered":"2023-10-14T11:58:15.155970+0000","last_unstale":"2023-10-14T11:58:16.299070+0000","last_undegraded":"2023-10-14T11:58:16.299070+0000","last_fullsized":"2023-10-14T11:58:16.299070+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:15:07.231894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.145083+0000","last_change":"2023-10-14T11:58:15.158350+0000","last_active":"2023-10-14T11:58:16.145083+0000","last_peered":"2023-10-14T11:58:16.145083+0000","last_clean":"2023-10-14T11:58:16.145083+0000","last_became_active":"2023-10-14T11:58:15.158036+0000","last_became_peered":"2023-10-14T11:58:15.158036+0000","last_unstale":"2023-10-14T11:58:16.145083+0000","last_undegraded":"2023-10-14T11:58:16.145083+0000","last_fullsized":"2023-10-14T11:58:16.145083+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:21:15.443882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144612+0000","last_change":"2023-10-14T11:58:15.160864+0000","last_active":"2023-10-14T11:58:16.144612+0000","last_peered":"2023-10-14T11:58:16.144612+0000","last_clean":"2023-10-14T11:58:16.144612+0000","last_became_active":"2023-10-14T11:58:15.157887+0000","last_became_peered":"2023-10-14T11:58:15.157887+0000","last_unstale":"2023-10-14T11:58:16.144612+0000","last_undegraded":"2023-10-14T11:58:16.144612+0000","last_fullsized":"2023-10-14T11:58:16.144612+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:23:17.281352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.299205+0000","last_change":"2023-10-14T11:58:15.148543+0000","last_active":"2023-10-14T11:58:16.299205+0000","last_peered":"2023-10-14T11:58:16.299205+0000","last_clean":"2023-10-14T11:58:16.299205+0000","last_became_active":"2023-10-14T11:58:15.148273+0000","last_became_peered":"2023-10-14T11:58:15.148273+0000","last_unstale":"2023-10-14T11:58:16.299205+0000","last_undegraded":"2023-10-14T11:58:16.299205+0000","last_fullsized":"2023-10-14T11:58:16.299205+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:35:58.025887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296512+0000","last_change":"2023-10-14T11:58:15.162159+0000","last_active":"2023-10-14T11:58:16.296512+0000","last_peered":"2023-10-14T11:58:16.296512+0000","last_clean":"2023-10-14T11:58:16.296512+0000","last_became_active":"2023-10-14T11:58:15.161859+0000","last_became_peered":"2023-10-14T11:58:15.161859+0000","last_unstale":"2023-10-14T11:58:16.296512+0000","last_undegraded":"2023-10-14T11:58:16.296512+0000","last_fullsized":"2023-10-14T11:58:16.296512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:59:03.464936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298805+0000","last_change":"2023-10-14T11:58:15.147368+0000","last_active":"2023-10-14T11:58:16.298805+0000","last_peered":"2023-10-14T11:58:16.298805+0000","last_clean":"2023-10-14T11:58:16.298805+0000","last_became_active":"2023-10-14T11:58:15.147083+0000","last_became_peered":"2023-10-14T11:58:15.147083+0000","last_unstale":"2023-10-14T11:58:16.298805+0000","last_undegraded":"2023-10-14T11:58:16.298805+0000","last_fullsized":"2023-10-14T11:58:16.298805+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:01:03.783116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296598+0000","last_change":"2023-10-14T11:58:15.170122+0000","last_active":"2023-10-14T11:58:16.296598+0000","last_peered":"2023-10-14T11:58:16.296598+0000","last_clean":"2023-10-14T11:58:16.296598+0000","last_became_active":"2023-10-14T11:58:15.169483+0000","last_became_peered":"2023-10-14T11:58:15.169483+0000","last_unstale":"2023-10-14T11:58:16.296598+0000","last_undegraded":"2023-10-14T11:58:16.296598+0000","last_fullsized":"2023-10-14T11:58:16.296598+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:15:07.231894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296034+0000","last_change":"2023-10-14T11:58:15.170506+0000","last_active":"2023-10-14T11:58:16.296034+0000","last_peered":"2023-10-14T11:58:16.296034+0000","last_clean":"2023-10-14T11:58:16.296034+0000","last_became_active":"2023-10-14T11:58:15.170243+0000","last_became_peered":"2023-10-14T11:58:15.170243+0000","last_unstale":"2023-10-14T11:58:16.296034+0000","last_undegraded":"2023-10-14T11:58:16.296034+0000","last_fullsized":"2023-10-14T11:58:16.296034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:30:51.306932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144532+0000","last_change":"2023-10-14T11:58:15.147517+0000","last_active":"2023-10-14T11:58:16.144532+0000","last_peered":"2023-10-14T11:58:16.144532+0000","last_clean":"2023-10-14T11:58:16.144532+0000","last_became_active":"2023-10-14T11:58:15.147245+0000","last_became_peered":"2023-10-14T11:58:15.147245+0000","last_unstale":"2023-10-14T11:58:16.144532+0000","last_undegraded":"2023-10-14T11:58:16.144532+0000","last_fullsized":"2023-10-14T11:58:16.144532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:21:15.443882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65244,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":286720,"data_stored":666519,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95799999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":75729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":666575,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65600000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":69269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27900000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78500000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0900000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1439999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T11:58:19.639 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-14T11:58:19.639 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-14T11:58:19.640 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-14T11:58:19.640 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-14T11:58:19.640 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-14T11:58:19.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.791+0000 7fe59e932700 1 Processor -- start 2023-10-14T11:58:19.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.791+0000 7fe59e932700 1 -- start start 2023-10-14T11:58:19.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.792+0000 7fe59e932700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe59806ab40 0x7fe59806f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:19.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.792+0000 7fe59e932700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe5980c0350 0x7fe59813d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:19.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.792+0000 7fe59e932700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe59806b630 con 0x7fe5981554e0 2023-10-14T11:58:19.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.792+0000 7fe59e932700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5980c49f0 con 0x7fe59806ab40 2023-10-14T11:58:19.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.792+0000 7fe59e932700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5980726a0 con 0x7fe5980c0350 2023-10-14T11:58:19.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.792+0000 7fe597fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe59806ab40 0x7fe59806f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:19.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.792+0000 7fe59cecf700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe5980c0350 0x7fe59813d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:19.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.792+0000 7fe5977fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fe5981554e0 0x7fe59806af20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.78:39194/0 (socket says 172.21.15.78:39194) 2023-10-14T11:58:19.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.792+0000 7fe59cecf700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe5980c0350 0x7fe59813d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.78:44190/0 (socket says 172.21.15.78:44190) 2023-10-14T11:58:19.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.792+0000 7fe597fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe59806ab40 0x7fe59806f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.78:47254/0 (socket says 172.21.15.78:47254) 2023-10-14T11:58:19.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.792+0000 7fe5977fe700 1 -- 172.21.15.78:0/2200330664 learned_addr learned my addr 172.21.15.78:0/2200330664 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:19.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.792+0000 7fe597fff700 1 -- 172.21.15.78:0/2200330664 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe5980c0350 msgr2=0x7fe59813d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:19.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.792+0000 7fe597fff700 1 --2- 172.21.15.78:0/2200330664 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe5980c0350 0x7fe59813d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:19.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.792+0000 7fe597fff700 1 -- 172.21.15.78:0/2200330664 >> v1:172.21.15.195:6789/0 conn(0x7fe5981554e0 legacy=0x7fe59806af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:19.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.792+0000 7fe597fff700 1 -- 172.21.15.78:0/2200330664 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe598072980 con 0x7fe59806ab40 2023-10-14T11:58:19.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.792+0000 7fe59cecf700 1 --2- 172.21.15.78:0/2200330664 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe5980c0350 0x7fe59813d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T11:58:19.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.793+0000 7fe597fff700 1 --2- 172.21.15.78:0/2200330664 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe59806ab40 0x7fe59806f1e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe58c009840 tx=0x7fe58c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f8c8954c8e7fca6 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:19.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.793+0000 7fe596ffd700 1 -- 172.21.15.78:0/2200330664 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe58c011830 con 0x7fe59806ab40 2023-10-14T11:58:19.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.793+0000 7fe596ffd700 1 -- 172.21.15.78:0/2200330664 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe58c011990 con 0x7fe59806ab40 2023-10-14T11:58:19.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.793+0000 7fe596ffd700 1 -- 172.21.15.78:0/2200330664 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe58c011d20 con 0x7fe59806ab40 2023-10-14T11:58:19.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.793+0000 7fe59e932700 1 -- 172.21.15.78:0/2200330664 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe59806ab40 msgr2=0x7fe59806f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:19.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.793+0000 7fe59e932700 1 --2- 172.21.15.78:0/2200330664 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe59806ab40 0x7fe59806f1e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe58c009840 tx=0x7fe58c009b50 comp rx=0 tx=0).stop 2023-10-14T11:58:19.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.793+0000 7fe59e932700 1 -- 172.21.15.78:0/2200330664 shutdown_connections 2023-10-14T11:58:19.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.793+0000 7fe59e932700 1 --2- 172.21.15.78:0/2200330664 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe5980c0350 0x7fe59813d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:19.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.793+0000 7fe59e932700 1 --2- 172.21.15.78:0/2200330664 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe59806ab40 0x7fe59806f1e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:19.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.793+0000 7fe59e932700 1 -- 172.21.15.78:0/2200330664 >> 172.21.15.78:0/2200330664 conn(0x7fe5980c3ac0 msgr2=0x7fe5980c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:19.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.793+0000 7fe59e932700 1 -- 172.21.15.78:0/2200330664 shutdown_connections 2023-10-14T11:58:19.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.793+0000 7fe59e932700 1 -- 172.21.15.78:0/2200330664 wait complete. 2023-10-14T11:58:19.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.795+0000 7fe59e932700 1 Processor -- start 2023-10-14T11:58:19.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.795+0000 7fe59e932700 1 -- start start 2023-10-14T11:58:19.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.795+0000 7fe59e932700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe59806ab40 0x7fe5981e9c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:19.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.795+0000 7fe59e932700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe5980c0350 0x7fe5981ec440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:19.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.795+0000 7fe59e932700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe5981554e0 0x7fe598150df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:19.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.795+0000 7fe59e932700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe598158c20 con 0x7fe59806ab40 2023-10-14T11:58:19.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.795+0000 7fe59e932700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5981357c0 con 0x7fe5980c0350 2023-10-14T11:58:19.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.795+0000 7fe59e932700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5981ea930 con 0x7fe5981554e0 2023-10-14T11:58:19.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.796+0000 7fe597fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe59806ab40 0x7fe5981e9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:19.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.796+0000 7fe597fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe59806ab40 0x7fe5981e9c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44522/0 (socket says 172.21.15.78:44522) 2023-10-14T11:58:19.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.796+0000 7fe597fff700 1 -- 172.21.15.78:0/1613357601 learned_addr learned my addr 172.21.15.78:0/1613357601 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:19.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.796+0000 7fe59cecf700 1 --2- 172.21.15.78:0/1613357601 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe5981554e0 0x7fe598150df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:19.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.796+0000 7fe5977fe700 1 --2- 172.21.15.78:0/1613357601 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe5980c0350 0x7fe5981ec440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:19.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.796+0000 7fe597fff700 1 -- 172.21.15.78:0/1613357601 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe5981554e0 msgr2=0x7fe598150df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:19.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.796+0000 7fe597fff700 1 --2- 172.21.15.78:0/1613357601 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe5981554e0 0x7fe598150df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:19.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.796+0000 7fe597fff700 1 -- 172.21.15.78:0/1613357601 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe5980c0350 msgr2=0x7fe5981ec440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:19.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.796+0000 7fe597fff700 1 --2- 172.21.15.78:0/1613357601 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe5980c0350 0x7fe5981ec440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:19.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.796+0000 7fe597fff700 1 -- 172.21.15.78:0/1613357601 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe590009a70 con 0x7fe59806ab40 2023-10-14T11:58:19.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.796+0000 7fe597fff700 1 --2- 172.21.15.78:0/1613357601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe59806ab40 0x7fe5981e9c60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe58c009fd0 tx=0x7fe58c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:19.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.796+0000 7fe594ff9700 1 -- 172.21.15.78:0/1613357601 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe58c0045d0 con 0x7fe59806ab40 2023-10-14T11:58:19.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.796+0000 7fe594ff9700 1 -- 172.21.15.78:0/1613357601 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe58c01e070 con 0x7fe59806ab40 2023-10-14T11:58:19.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.796+0000 7fe59e932700 1 -- 172.21.15.78:0/1613357601 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe58c0094a0 con 0x7fe59806ab40 2023-10-14T11:58:19.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.796+0000 7fe594ff9700 1 -- 172.21.15.78:0/1613357601 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe58c0038e0 con 0x7fe59806ab40 2023-10-14T11:58:19.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.796+0000 7fe59e932700 1 -- 172.21.15.78:0/1613357601 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5981ea650 con 0x7fe59806ab40 2023-10-14T11:58:19.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.797+0000 7fe59e932700 1 -- 172.21.15.78:0/1613357601 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe598074b80 con 0x7fe59806ab40 2023-10-14T11:58:19.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.798+0000 7fe594ff9700 1 -- 172.21.15.78:0/1613357601 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe58c003a40 con 0x7fe59806ab40 2023-10-14T11:58:19.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.798+0000 7fe594ff9700 1 --2- 172.21.15.78:0/1613357601 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe58406af00 0x7fe58406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:19.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.798+0000 7fe594ff9700 1 -- 172.21.15.78:0/1613357601 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe58c08a390 con 0x7fe59806ab40 2023-10-14T11:58:19.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.798+0000 7fe5977fe700 1 --2- 172.21.15.78:0/1613357601 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe58406af00 0x7fe58406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:19.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.798+0000 7fe5977fe700 1 --2- 172.21.15.78:0/1613357601 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe58406af00 0x7fe58406d3c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe59814ec00 tx=0x7fe588009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:19.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:19.802+0000 7fe594ff9700 1 -- 172.21.15.78:0/1613357601 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe58c0554d0 con 0x7fe59806ab40 2023-10-14T11:58:20.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.051+0000 7fe59e932700 1 -- 172.21.15.78:0/1613357601 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fe598150a10 con 0x7fe59806ab40 2023-10-14T11:58:20.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.054+0000 7fe594ff9700 1 -- 172.21.15.78:0/1613357601 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171028 (secure 0 0 0) 0x7fe58c058af0 con 0x7fe59806ab40 2023-10-14T11:58:20.058 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:20.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.057+0000 7fe59e932700 1 -- 172.21.15.78:0/1613357601 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe58406af00 msgr2=0x7fe58406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:20.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.057+0000 7fe59e932700 1 --2- 172.21.15.78:0/1613357601 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe58406af00 0x7fe58406d3c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe59814ec00 tx=0x7fe588009040 comp rx=0 tx=0).stop 2023-10-14T11:58:20.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.057+0000 7fe59e932700 1 -- 172.21.15.78:0/1613357601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe59806ab40 msgr2=0x7fe5981e9c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:20.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.057+0000 7fe59e932700 1 --2- 172.21.15.78:0/1613357601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe59806ab40 0x7fe5981e9c60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe58c009fd0 tx=0x7fe58c004750 comp rx=0 tx=0).stop 2023-10-14T11:58:20.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.057+0000 7fe59e932700 1 -- 172.21.15.78:0/1613357601 shutdown_connections 2023-10-14T11:58:20.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.057+0000 7fe59e932700 1 --2- 172.21.15.78:0/1613357601 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe5981554e0 0x7fe598150df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.057+0000 7fe59e932700 1 --2- 172.21.15.78:0/1613357601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe59806ab40 0x7fe5981e9c60 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.057+0000 7fe59e932700 1 --2- 172.21.15.78:0/1613357601 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe58406af00 0x7fe58406d3c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.057+0000 7fe59e932700 1 --2- 172.21.15.78:0/1613357601 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe5980c0350 0x7fe5981ec440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.057+0000 7fe59e932700 1 -- 172.21.15.78:0/1613357601 >> 172.21.15.78:0/1613357601 conn(0x7fe5980c3ac0 msgr2=0x7fe598153270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:20.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.057+0000 7fe59e932700 1 -- 172.21.15.78:0/1613357601 shutdown_connections 2023-10-14T11:58:20.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.057+0000 7fe59e932700 1 -- 172.21.15.78:0/1613357601 wait complete. 2023-10-14T11:58:20.071 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":5,"active_gid":4104,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":3171890074},{"type":"v1","addr":"172.21.15.78:6833","nonce":3171890074}]},"active_addr":"172.21.15.78:6833/3171890074","active_change":"2023-10-14T11:58:02.950068+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4100,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":3627417149}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":209757725}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":3441212569}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":727060608}]}]} 2023-10-14T11:58:20.076 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-14T11:58:20.076 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-14T11:58:20.076 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T11:58:20.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.195+0000 7fe1059be700 1 Processor -- start 2023-10-14T11:58:20.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.195+0000 7fe1059be700 1 -- start start 2023-10-14T11:58:20.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.195+0000 7fe1059be700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe10014b7e0 0x7fe1000c5150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:20.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.195+0000 7fe1059be700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe100154b90 0x7fe1000c9260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:20.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.195+0000 7fe1059be700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1000c4e40 con 0x7fe1000c97a0 2023-10-14T11:58:20.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.195+0000 7fe1059be700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1000c81c0 con 0x7fe10014b7e0 2023-10-14T11:58:20.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.195+0000 7fe1059be700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1000c84a0 con 0x7fe100154b90 2023-10-14T11:58:20.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.196+0000 7fe0ff7fe700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fe1000c97a0 0x7fe10006b440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49406/0 (socket says 172.21.15.78:49406) 2023-10-14T11:58:20.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.196+0000 7fe0ff7fe700 1 -- 172.21.15.78:0/3526227543 learned_addr learned my addr 172.21.15.78:0/3526227543 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:20.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.196+0000 7fe0fe7fc700 1 --2- 172.21.15.78:0/3526227543 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe100154b90 0x7fe1000c9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:20.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.196+0000 7fe0feffd700 1 --2- 172.21.15.78:0/3526227543 >> v2:172.21.15.195:3300/0 conn(0x7fe10014b7e0 0x7fe1000c5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:20.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.196+0000 7fe0fdffb700 1 -- 172.21.15.78:0/3526227543 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3854632291 0 0) 0x7fe1000c4e40 con 0x7fe1000c97a0 2023-10-14T11:58:20.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.196+0000 7fe0fdffb700 1 -- 172.21.15.78:0/3526227543 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0e80035b0 con 0x7fe1000c97a0 2023-10-14T11:58:20.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.196+0000 7fe0fdffb700 1 -- 172.21.15.78:0/3526227543 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3218297041 0 0) 0x7fe0e80035b0 con 0x7fe1000c97a0 2023-10-14T11:58:20.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.196+0000 7fe0fdffb700 1 -- 172.21.15.78:0/3526227543 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe100154b90 msgr2=0x7fe1000c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:20.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.196+0000 7fe0fdffb700 1 --2- 172.21.15.78:0/3526227543 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe100154b90 0x7fe1000c9260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.196+0000 7fe0fdffb700 1 -- 172.21.15.78:0/3526227543 >> v2:172.21.15.195:3300/0 conn(0x7fe10014b7e0 msgr2=0x7fe1000c5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:20.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.196+0000 7fe0fdffb700 1 --2- 172.21.15.78:0/3526227543 >> v2:172.21.15.195:3300/0 conn(0x7fe10014b7e0 0x7fe1000c5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.196+0000 7fe0fdffb700 1 -- 172.21.15.78:0/3526227543 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1000c8780 con 0x7fe1000c97a0 2023-10-14T11:58:20.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.196+0000 7fe0fdffb700 1 -- 172.21.15.78:0/3526227543 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441799917 0 0) 0x7fe0f80019f0 con 0x7fe1000c97a0 2023-10-14T11:58:20.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.196+0000 7fe0fdffb700 1 -- 172.21.15.78:0/3526227543 >> v1:172.21.15.78:6789/0 conn(0x7fe1000c97a0 legacy=0x7fe10006b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:20.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.196+0000 7fe0fdffb700 1 --2- 172.21.15.78:0/3526227543 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe100154b90 0x7fe0e8003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:20.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.196+0000 7fe0fdffb700 1 --2- 172.21.15.78:0/3526227543 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe0e8003f80 0x7fe0e8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:20.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.196+0000 7fe0fdffb700 1 --2- 172.21.15.78:0/3526227543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0e8004d90 0x7fe0e8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:20.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.196+0000 7fe0fdffb700 1 -- 172.21.15.78:0/3526227543 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1000c81c0 con 0x7fe0e8004d90 2023-10-14T11:58:20.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.196+0000 7fe0fdffb700 1 -- 172.21.15.78:0/3526227543 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1000c84a0 con 0x7fe0e8003f80 2023-10-14T11:58:20.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.196+0000 7fe0fdffb700 1 -- 172.21.15.78:0/3526227543 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e800ba60 con 0x7fe100154b90 2023-10-14T11:58:20.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.196+0000 7fe0ff7fe700 1 --2- 172.21.15.78:0/3526227543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0e8004d90 0x7fe0e8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:20.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.196+0000 7fe0fe7fc700 1 --2- 172.21.15.78:0/3526227543 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe0e8003f80 0x7fe0e8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:20.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.196+0000 7fe0feffd700 1 --2- 172.21.15.78:0/3526227543 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe100154b90 0x7fe0e8003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:20.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.196+0000 7fe0ff7fe700 1 -- 172.21.15.78:0/3526227543 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe100154b90 msgr2=0x7fe0e8003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:20.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.196+0000 7fe0ff7fe700 1 --2- 172.21.15.78:0/3526227543 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe100154b90 0x7fe0e8003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.196+0000 7fe0ff7fe700 1 -- 172.21.15.78:0/3526227543 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe0e8003f80 msgr2=0x7fe0e8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:20.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.196+0000 7fe0ff7fe700 1 --2- 172.21.15.78:0/3526227543 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe0e8003f80 0x7fe0e8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.196+0000 7fe0ff7fe700 1 -- 172.21.15.78:0/3526227543 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e80035b0 con 0x7fe0e8004d90 2023-10-14T11:58:20.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.196+0000 7fe0feffd700 1 --2- 172.21.15.78:0/3526227543 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe100154b90 0x7fe0e8003a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:20.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.197+0000 7fe0ff7fe700 1 --2- 172.21.15.78:0/3526227543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0e8004d90 0x7fe0e8005210 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe10008bf00 tx=0x7fe0e8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f63f2dce02b77645 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:20.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.197+0000 7fe0fdffb700 1 -- 172.21.15.78:0/3526227543 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f800a360 con 0x7fe0e8004d90 2023-10-14T11:58:20.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.197+0000 7fe0fdffb700 1 -- 172.21.15.78:0/3526227543 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0f8012600 con 0x7fe0e8004d90 2023-10-14T11:58:20.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.197+0000 7fe0fdffb700 1 -- 172.21.15.78:0/3526227543 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f80129b0 con 0x7fe0e8004d90 2023-10-14T11:58:20.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.197+0000 7fe1059be700 1 -- 172.21.15.78:0/3526227543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0e8004d90 msgr2=0x7fe0e8005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:20.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.197+0000 7fe1059be700 1 --2- 172.21.15.78:0/3526227543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0e8004d90 0x7fe0e8005210 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe10008bf00 tx=0x7fe0e8005880 comp rx=0 tx=0).stop 2023-10-14T11:58:20.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.197+0000 7fe0ff7fe700 1 -- 172.21.15.78:0/3526227543 reap_dead start 2023-10-14T11:58:20.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.197+0000 7fe1059be700 1 -- 172.21.15.78:0/3526227543 shutdown_connections 2023-10-14T11:58:20.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.197+0000 7fe1059be700 1 -- 172.21.15.78:0/3526227543 >> 172.21.15.78:0/3526227543 conn(0x7fe1000ca2c0 msgr2=0x7fe1000ca6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:20.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.197+0000 7fe1059be700 1 -- 172.21.15.78:0/3526227543 shutdown_connections 2023-10-14T11:58:20.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.197+0000 7fe1059be700 1 -- 172.21.15.78:0/3526227543 wait complete. 2023-10-14T11:58:20.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.199+0000 7fe1059be700 1 Processor -- start 2023-10-14T11:58:20.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.199+0000 7fe1059be700 1 -- start start 2023-10-14T11:58:20.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.200+0000 7fe1059be700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe0e8004d90 0x7fe100144800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:20.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.200+0000 7fe1059be700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe10014d800 0x7fe1000ab430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:20.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.200+0000 7fe1059be700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe10014c450 0x7fe10014c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:20.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.200+0000 7fe1059be700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1001eb060 con 0x7fe10014d800 2023-10-14T11:58:20.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.200+0000 7fe1059be700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe10014ab80 con 0x7fe0e8004d90 2023-10-14T11:58:20.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.200+0000 7fe1059be700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe10014bc90 con 0x7fe10014c450 2023-10-14T11:58:20.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.200+0000 7fe0fe7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe10014d800 0x7fe1000ab430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:20.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.200+0000 7fe0fe7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe10014d800 0x7fe1000ab430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44540/0 (socket says 172.21.15.78:44540) 2023-10-14T11:58:20.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.200+0000 7fe0fe7fc700 1 -- 172.21.15.78:0/2295721106 learned_addr learned my addr 172.21.15.78:0/2295721106 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:20.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.200+0000 7fe0ff7fe700 1 --2- 172.21.15.78:0/2295721106 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe10014c450 0x7fe10014c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:20.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.200+0000 7fe0fe7fc700 1 -- 172.21.15.78:0/2295721106 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe10014c450 msgr2=0x7fe10014c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:20.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.200+0000 7fe0fe7fc700 1 --2- 172.21.15.78:0/2295721106 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe10014c450 0x7fe10014c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.200+0000 7fe0fe7fc700 1 -- 172.21.15.78:0/2295721106 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe0e8004d90 msgr2=0x7fe100144800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:20.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.200+0000 7fe0fe7fc700 1 --2- 172.21.15.78:0/2295721106 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe0e8004d90 0x7fe100144800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.200+0000 7fe0fe7fc700 1 -- 172.21.15.78:0/2295721106 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1001eb250 con 0x7fe10014d800 2023-10-14T11:58:20.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.200+0000 7fe0fe7fc700 1 --2- 172.21.15.78:0/2295721106 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe10014d800 0x7fe1000ab430 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe0f4011930 tx=0x7fe0f4011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:20.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.200+0000 7fe0e7fff700 1 -- 172.21.15.78:0/2295721106 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f400f940 con 0x7fe10014d800 2023-10-14T11:58:20.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.201+0000 7fe0e7fff700 1 -- 172.21.15.78:0/2295721106 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0f400faa0 con 0x7fe10014d800 2023-10-14T11:58:20.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.201+0000 7fe1059be700 1 -- 172.21.15.78:0/2295721106 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe10014bf30 con 0x7fe10014d800 2023-10-14T11:58:20.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.201+0000 7fe0e7fff700 1 -- 172.21.15.78:0/2295721106 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f400fe30 con 0x7fe10014d800 2023-10-14T11:58:20.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.201+0000 7fe1059be700 1 -- 172.21.15.78:0/2295721106 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe100144d70 con 0x7fe10014d800 2023-10-14T11:58:20.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.202+0000 7fe1059be700 1 -- 172.21.15.78:0/2295721106 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1000a9a80 con 0x7fe10014d800 2023-10-14T11:58:20.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.202+0000 7fe0e7fff700 1 -- 172.21.15.78:0/2295721106 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe0f4014440 con 0x7fe10014d800 2023-10-14T11:58:20.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.202+0000 7fe0e7fff700 1 --2- 172.21.15.78:0/2295721106 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe0ec06af00 0x7fe0ec06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:20.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.202+0000 7fe0e7fff700 1 -- 172.21.15.78:0/2295721106 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe0f4089cd0 con 0x7fe10014d800 2023-10-14T11:58:20.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.202+0000 7fe0feffd700 1 --2- 172.21.15.78:0/2295721106 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe0ec06af00 0x7fe0ec06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:20.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.203+0000 7fe0feffd700 1 --2- 172.21.15.78:0/2295721106 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe0ec06af00 0x7fe0ec06d3c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe1001524f0 tx=0x7fe0f0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:20.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.207+0000 7fe0e7fff700 1 -- 172.21.15.78:0/2295721106 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe0f4054e20 con 0x7fe10014d800 2023-10-14T11:58:20.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.398+0000 7fe1059be700 1 -- 172.21.15.78:0/2295721106 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe100074b80 con 0x7fe10014d800 2023-10-14T11:58:20.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.399+0000 7fe0e7fff700 1 -- 172.21.15.78:0/2295721106 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16481 (secure 0 0 0) 0x7fe0f4058440 con 0x7fe10014d800 2023-10-14T11:58:20.401 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:20.401 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":21,"fsid":"4c5dc134-3092-406a-a1e1-13e6d9e926e8","created":"2023-10-14T11:57:57.363750+0000","modified":"2023-10-14T11:58:16.139413+0000","last_up_change":"2023-10-14T11:58:06.982472+0000","last_in_change":"2023-10-14T11:57:59.821373+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T11:58:09.001200+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T11:58:12.778573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T11:58:13.490286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"625a5fc0-9474-4eae-b80e-14c27a74ad19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6801","nonce":3884969780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6803","nonce":3884969780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6807","nonce":3884969780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6805","nonce":3884969780}]},"public_addr":"172.21.15.78:6801/3884969780","cluster_addr":"172.21.15.78:6803/3884969780","heartbeat_back_addr":"172.21.15.78:6807/3884969780","heartbeat_front_addr":"172.21.15.78:6805/3884969780","state":["exists","up"]},{"osd":1,"uuid":"49d63db1-2981-49ab-9c40-c50fc744b982","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6809","nonce":2176928406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6811","nonce":2176928406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6815","nonce":2176928406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6813","nonce":2176928406}]},"public_addr":"172.21.15.78:6809/2176928406","cluster_addr":"172.21.15.78:6811/2176928406","heartbeat_back_addr":"172.21.15.78:6815/2176928406","heartbeat_front_addr":"172.21.15.78:6813/2176928406","state":["exists","up"]},{"osd":2,"uuid":"1d7d0b29-cbd1-4717-9fe6-765a8d4998b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6825","nonce":3647044577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6827","nonce":3647044577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6831","nonce":3647044577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6829","nonce":3647044577}]},"public_addr":"172.21.15.78:6825/3647044577","cluster_addr":"172.21.15.78:6827/3647044577","heartbeat_back_addr":"172.21.15.78:6831/3647044577","heartbeat_front_addr":"172.21.15.78:6829/3647044577","state":["exists","up"]},{"osd":3,"uuid":"2f75ded1-e9e2-40e7-99ae-45e1e425e1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6817","nonce":3584112288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6819","nonce":3584112288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6823","nonce":3584112288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6821","nonce":3584112288}]},"public_addr":"172.21.15.78:6817/3584112288","cluster_addr":"172.21.15.78:6819/3584112288","heartbeat_back_addr":"172.21.15.78:6823/3584112288","heartbeat_front_addr":"172.21.15.78:6821/3584112288","state":["exists","up"]},{"osd":4,"uuid":"5130b432-8358-46cb-8a31-3064c976f6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6801","nonce":3732756301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6803","nonce":3732756301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6807","nonce":3732756301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6805","nonce":3732756301}]},"public_addr":"172.21.15.195:6801/3732756301","cluster_addr":"172.21.15.195:6803/3732756301","heartbeat_back_addr":"172.21.15.195:6807/3732756301","heartbeat_front_addr":"172.21.15.195:6805/3732756301","state":["exists","up"]},{"osd":5,"uuid":"60e234a9-a506-4e40-9c19-49a58f3cf28c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6825","nonce":2857706411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6827","nonce":2857706411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6831","nonce":2857706411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6829","nonce":2857706411}]},"public_addr":"172.21.15.195:6825/2857706411","cluster_addr":"172.21.15.195:6827/2857706411","heartbeat_back_addr":"172.21.15.195:6831/2857706411","heartbeat_front_addr":"172.21.15.195:6829/2857706411","state":["exists","up"]},{"osd":6,"uuid":"df7d2dd6-26b6-4eb1-96e5-94d1f3955b33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6810","nonce":72554374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6813","nonce":72554374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6820","nonce":72554374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6817","nonce":72554374}]},"public_addr":"172.21.15.195:6810/72554374","cluster_addr":"172.21.15.195:6813/72554374","heartbeat_back_addr":"172.21.15.195:6820/72554374","heartbeat_front_addr":"172.21.15.195:6817/72554374","state":["exists","up"]},{"osd":7,"uuid":"47a9d4de-f56d-4aa4-9900-c4a9cf9b2f97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6809","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6811","nonce":482472979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6816","nonce":482472979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6823","nonce":482472979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6821","nonce":482472979}]},"public_addr":"172.21.15.195:6811/482472979","cluster_addr":"172.21.15.195:6816/482472979","heartbeat_back_addr":"172.21.15.195:6823/482472979","heartbeat_front_addr":"172.21.15.195:6821/482472979","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.754003+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.005994+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.043603+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.024506+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.062585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.033767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.994825+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.038300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T11:58:20.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.401+0000 7fe1059be700 1 -- 172.21.15.78:0/2295721106 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe0ec06af00 msgr2=0x7fe0ec06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:20.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.401+0000 7fe1059be700 1 --2- 172.21.15.78:0/2295721106 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe0ec06af00 0x7fe0ec06d3c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe1001524f0 tx=0x7fe0f0008040 comp rx=0 tx=0).stop 2023-10-14T11:58:20.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.401+0000 7fe1059be700 1 -- 172.21.15.78:0/2295721106 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe10014d800 msgr2=0x7fe1000ab430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:20.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.401+0000 7fe1059be700 1 --2- 172.21.15.78:0/2295721106 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe10014d800 0x7fe1000ab430 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe0f4011930 tx=0x7fe0f4011cf0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.401+0000 7fe1059be700 1 -- 172.21.15.78:0/2295721106 shutdown_connections 2023-10-14T11:58:20.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.401+0000 7fe1059be700 1 --2- 172.21.15.78:0/2295721106 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe10014c450 0x7fe10014c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.401+0000 7fe1059be700 1 --2- 172.21.15.78:0/2295721106 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe0ec06af00 0x7fe0ec06d3c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.401+0000 7fe1059be700 1 --2- 172.21.15.78:0/2295721106 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe0e8004d90 0x7fe100144800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.401+0000 7fe1059be700 1 --2- 172.21.15.78:0/2295721106 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe10014d800 0x7fe1000ab430 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.401+0000 7fe1059be700 1 -- 172.21.15.78:0/2295721106 >> 172.21.15.78:0/2295721106 conn(0x7fe1000ca2c0 msgr2=0x7fe1000c6050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:20.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.401+0000 7fe1059be700 1 -- 172.21.15.78:0/2295721106 shutdown_connections 2023-10-14T11:58:20.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.401+0000 7fe1059be700 1 -- 172.21.15.78:0/2295721106 wait complete. 2023-10-14T11:58:20.415 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-14T11:58:20.415 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T11:58:20.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.533+0000 7f846c4f7700 1 Processor -- start 2023-10-14T11:58:20.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.533+0000 7f846c4f7700 1 -- start start 2023-10-14T11:58:20.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.534+0000 7f846c4f7700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f846406ab40 0x7f846406f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:20.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.534+0000 7f846c4f7700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f84641554e0 0x7f846406af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:20.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.534+0000 7f846c4f7700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f84640c0350 0x7f846413d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:20.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.534+0000 7f846c4f7700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84640c49f0 con 0x7f84640c0350 2023-10-14T11:58:20.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.534+0000 7f846c4f7700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84640726a0 con 0x7f846406ab40 2023-10-14T11:58:20.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.534+0000 7f846c4f7700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8464072980 con 0x7f84641554e0 2023-10-14T11:58:20.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.534+0000 7f846aa94700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f84640c0350 0x7f846413d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:20.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.534+0000 7f846aa94700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f84640c0350 0x7f846413d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44550/0 (socket says 172.21.15.78:44550) 2023-10-14T11:58:20.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.534+0000 7f846aa94700 1 -- 172.21.15.78:0/3482959240 learned_addr learned my addr 172.21.15.78:0/3482959240 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:20.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.534+0000 7f8469a92700 1 --2- 172.21.15.78:0/3482959240 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f84641554e0 0x7f846406af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:20.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.534+0000 7f846a293700 1 --2- 172.21.15.78:0/3482959240 >> v2:172.21.15.195:3300/0 conn(0x7f846406ab40 0x7f846406f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:20.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.534+0000 7f846aa94700 1 -- 172.21.15.78:0/3482959240 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f84641554e0 msgr2=0x7f846406af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:20.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.534+0000 7f846aa94700 1 --2- 172.21.15.78:0/3482959240 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f84641554e0 0x7f846406af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.534+0000 7f846aa94700 1 -- 172.21.15.78:0/3482959240 >> v2:172.21.15.195:3300/0 conn(0x7f846406ab40 msgr2=0x7f846406f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:20.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.534+0000 7f846aa94700 1 --2- 172.21.15.78:0/3482959240 >> v2:172.21.15.195:3300/0 conn(0x7f846406ab40 0x7f846406f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.534+0000 7f846aa94700 1 -- 172.21.15.78:0/3482959240 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8464154a00 con 0x7f84640c0350 2023-10-14T11:58:20.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.534+0000 7f846aa94700 1 --2- 172.21.15.78:0/3482959240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84640c0350 0x7f846413d7b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8464072bc0 tx=0x7f845c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d33bb8801eb03e29 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:20.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.535+0000 7f8469291700 1 -- 172.21.15.78:0/3482959240 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f845c00e070 con 0x7f84640c0350 2023-10-14T11:58:20.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.535+0000 7f8469291700 1 -- 172.21.15.78:0/3482959240 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f845c00b9c0 con 0x7f84640c0350 2023-10-14T11:58:20.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.535+0000 7f8469291700 1 -- 172.21.15.78:0/3482959240 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f845c00bd70 con 0x7f84640c0350 2023-10-14T11:58:20.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.535+0000 7f846c4f7700 1 -- 172.21.15.78:0/3482959240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84640c0350 msgr2=0x7f846413d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:20.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.535+0000 7f846c4f7700 1 --2- 172.21.15.78:0/3482959240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84640c0350 0x7f846413d7b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8464072bc0 tx=0x7f845c00b330 comp rx=0 tx=0).stop 2023-10-14T11:58:20.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.535+0000 7f846c4f7700 1 -- 172.21.15.78:0/3482959240 shutdown_connections 2023-10-14T11:58:20.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.535+0000 7f846c4f7700 1 --2- 172.21.15.78:0/3482959240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84640c0350 0x7f846413d7b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.535+0000 7f846c4f7700 1 --2- 172.21.15.78:0/3482959240 >> v2:172.21.15.195:3300/0 conn(0x7f846406ab40 0x7f846406f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.535+0000 7f846c4f7700 1 --2- 172.21.15.78:0/3482959240 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f84641554e0 0x7f846406af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.535+0000 7f846c4f7700 1 -- 172.21.15.78:0/3482959240 >> 172.21.15.78:0/3482959240 conn(0x7f84640c3ac0 msgr2=0x7f84640c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:20.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.535+0000 7f846c4f7700 1 -- 172.21.15.78:0/3482959240 shutdown_connections 2023-10-14T11:58:20.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.535+0000 7f846c4f7700 1 -- 172.21.15.78:0/3482959240 wait complete. 2023-10-14T11:58:20.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.536+0000 7f846c4f7700 1 Processor -- start 2023-10-14T11:58:20.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.537+0000 7f846c4f7700 1 -- start start 2023-10-14T11:58:20.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.537+0000 7f846c4f7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f846406ab40 0x7f84641ed360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:20.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.537+0000 7f846c4f7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84640c0350 0x7f84640c6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:20.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.537+0000 7f846c4f7700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f84641554e0 0x7f8464150020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:20.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.537+0000 7f846c4f7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84641517f0 con 0x7f846406ab40 2023-10-14T11:58:20.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.537+0000 7f846c4f7700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8464150d00 con 0x7f84640c0350 2023-10-14T11:58:20.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.537+0000 7f846c4f7700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84640a86c0 con 0x7f84641554e0 2023-10-14T11:58:20.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.537+0000 7f846a293700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f846406ab40 0x7f84641ed360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:20.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.537+0000 7f846a293700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f846406ab40 0x7f84641ed360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44552/0 (socket says 172.21.15.78:44552) 2023-10-14T11:58:20.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.537+0000 7f846a293700 1 -- 172.21.15.78:0/1524599085 learned_addr learned my addr 172.21.15.78:0/1524599085 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:20.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.537+0000 7f8469a92700 1 --2- 172.21.15.78:0/1524599085 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84640c0350 0x7f84640c6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:20.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.537+0000 7f846aa94700 1 --2- 172.21.15.78:0/1524599085 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f84641554e0 0x7f8464150020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:20.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.538+0000 7f846a293700 1 -- 172.21.15.78:0/1524599085 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f84641554e0 msgr2=0x7f8464150020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:20.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.538+0000 7f846a293700 1 --2- 172.21.15.78:0/1524599085 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f84641554e0 0x7f8464150020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.538+0000 7f846a293700 1 -- 172.21.15.78:0/1524599085 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84640c0350 msgr2=0x7f84640c6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:20.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.538+0000 7f846a293700 1 --2- 172.21.15.78:0/1524599085 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84640c0350 0x7f84640c6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.538+0000 7f846a293700 1 -- 172.21.15.78:0/1524599085 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f845c009b30 con 0x7f846406ab40 2023-10-14T11:58:20.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.538+0000 7f846a293700 1 --2- 172.21.15.78:0/1524599085 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f846406ab40 0x7f84641ed360 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f845400e810 tx=0x7f845400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:20.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.538+0000 7f845affd700 1 -- 172.21.15.78:0/1524599085 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f845400c070 con 0x7f846406ab40 2023-10-14T11:58:20.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.538+0000 7f845affd700 1 -- 172.21.15.78:0/1524599085 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8454013610 con 0x7f846406ab40 2023-10-14T11:58:20.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.538+0000 7f846c4f7700 1 -- 172.21.15.78:0/1524599085 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84641ed8a0 con 0x7f846406ab40 2023-10-14T11:58:20.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.538+0000 7f845affd700 1 -- 172.21.15.78:0/1524599085 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84540139c0 con 0x7f846406ab40 2023-10-14T11:58:20.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.538+0000 7f846c4f7700 1 -- 172.21.15.78:0/1524599085 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8464151230 con 0x7f846406ab40 2023-10-14T11:58:20.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.539+0000 7f846c4f7700 1 -- 172.21.15.78:0/1524599085 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8464074b80 con 0x7f846406ab40 2023-10-14T11:58:20.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.539+0000 7f845affd700 1 -- 172.21.15.78:0/1524599085 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8454013b20 con 0x7f846406ab40 2023-10-14T11:58:20.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.540+0000 7f845affd700 1 --2- 172.21.15.78:0/1524599085 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f844806af00 0x7f844806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:20.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.540+0000 7f8469a92700 1 --2- 172.21.15.78:0/1524599085 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f844806af00 0x7f844806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:20.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.540+0000 7f845affd700 1 -- 172.21.15.78:0/1524599085 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f84540893b0 con 0x7f846406ab40 2023-10-14T11:58:20.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.540+0000 7f8469a92700 1 --2- 172.21.15.78:0/1524599085 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f844806af00 0x7f844806d3c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f84641e71e0 tx=0x7f8460008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:20.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.544+0000 7f845affd700 1 -- 172.21.15.78:0/1524599085 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8454054470 con 0x7f846406ab40 2023-10-14T11:58:20.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.734+0000 7f846c4f7700 1 -- 172.21.15.78:0/1524599085 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84640a8490 con 0x7f846406ab40 2023-10-14T11:58:20.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.735+0000 7f845affd700 1 -- 172.21.15.78:0/1524599085 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16481 (secure 0 0 0) 0x7f8454057a90 con 0x7f846406ab40 2023-10-14T11:58:20.737 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:20.737 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":21,"fsid":"4c5dc134-3092-406a-a1e1-13e6d9e926e8","created":"2023-10-14T11:57:57.363750+0000","modified":"2023-10-14T11:58:16.139413+0000","last_up_change":"2023-10-14T11:58:06.982472+0000","last_in_change":"2023-10-14T11:57:59.821373+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T11:58:09.001200+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T11:58:12.778573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T11:58:13.490286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"625a5fc0-9474-4eae-b80e-14c27a74ad19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6801","nonce":3884969780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6803","nonce":3884969780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6807","nonce":3884969780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6805","nonce":3884969780}]},"public_addr":"172.21.15.78:6801/3884969780","cluster_addr":"172.21.15.78:6803/3884969780","heartbeat_back_addr":"172.21.15.78:6807/3884969780","heartbeat_front_addr":"172.21.15.78:6805/3884969780","state":["exists","up"]},{"osd":1,"uuid":"49d63db1-2981-49ab-9c40-c50fc744b982","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6809","nonce":2176928406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6811","nonce":2176928406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6815","nonce":2176928406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6813","nonce":2176928406}]},"public_addr":"172.21.15.78:6809/2176928406","cluster_addr":"172.21.15.78:6811/2176928406","heartbeat_back_addr":"172.21.15.78:6815/2176928406","heartbeat_front_addr":"172.21.15.78:6813/2176928406","state":["exists","up"]},{"osd":2,"uuid":"1d7d0b29-cbd1-4717-9fe6-765a8d4998b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6825","nonce":3647044577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6827","nonce":3647044577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6831","nonce":3647044577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6829","nonce":3647044577}]},"public_addr":"172.21.15.78:6825/3647044577","cluster_addr":"172.21.15.78:6827/3647044577","heartbeat_back_addr":"172.21.15.78:6831/3647044577","heartbeat_front_addr":"172.21.15.78:6829/3647044577","state":["exists","up"]},{"osd":3,"uuid":"2f75ded1-e9e2-40e7-99ae-45e1e425e1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6817","nonce":3584112288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6819","nonce":3584112288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6823","nonce":3584112288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6821","nonce":3584112288}]},"public_addr":"172.21.15.78:6817/3584112288","cluster_addr":"172.21.15.78:6819/3584112288","heartbeat_back_addr":"172.21.15.78:6823/3584112288","heartbeat_front_addr":"172.21.15.78:6821/3584112288","state":["exists","up"]},{"osd":4,"uuid":"5130b432-8358-46cb-8a31-3064c976f6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6801","nonce":3732756301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6803","nonce":3732756301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6807","nonce":3732756301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6805","nonce":3732756301}]},"public_addr":"172.21.15.195:6801/3732756301","cluster_addr":"172.21.15.195:6803/3732756301","heartbeat_back_addr":"172.21.15.195:6807/3732756301","heartbeat_front_addr":"172.21.15.195:6805/3732756301","state":["exists","up"]},{"osd":5,"uuid":"60e234a9-a506-4e40-9c19-49a58f3cf28c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6825","nonce":2857706411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6827","nonce":2857706411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6831","nonce":2857706411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6829","nonce":2857706411}]},"public_addr":"172.21.15.195:6825/2857706411","cluster_addr":"172.21.15.195:6827/2857706411","heartbeat_back_addr":"172.21.15.195:6831/2857706411","heartbeat_front_addr":"172.21.15.195:6829/2857706411","state":["exists","up"]},{"osd":6,"uuid":"df7d2dd6-26b6-4eb1-96e5-94d1f3955b33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6810","nonce":72554374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6813","nonce":72554374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6820","nonce":72554374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6817","nonce":72554374}]},"public_addr":"172.21.15.195:6810/72554374","cluster_addr":"172.21.15.195:6813/72554374","heartbeat_back_addr":"172.21.15.195:6820/72554374","heartbeat_front_addr":"172.21.15.195:6817/72554374","state":["exists","up"]},{"osd":7,"uuid":"47a9d4de-f56d-4aa4-9900-c4a9cf9b2f97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6809","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6811","nonce":482472979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6816","nonce":482472979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6823","nonce":482472979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6821","nonce":482472979}]},"public_addr":"172.21.15.195:6811/482472979","cluster_addr":"172.21.15.195:6816/482472979","heartbeat_back_addr":"172.21.15.195:6823/482472979","heartbeat_front_addr":"172.21.15.195:6821/482472979","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.754003+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.005994+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.043603+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.024506+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.062585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.033767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.994825+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.038300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T11:58:20.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.737+0000 7f846c4f7700 1 -- 172.21.15.78:0/1524599085 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f844806af00 msgr2=0x7f844806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:20.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.737+0000 7f846c4f7700 1 --2- 172.21.15.78:0/1524599085 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f844806af00 0x7f844806d3c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f84641e71e0 tx=0x7f8460008040 comp rx=0 tx=0).stop 2023-10-14T11:58:20.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.737+0000 7f846c4f7700 1 -- 172.21.15.78:0/1524599085 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f846406ab40 msgr2=0x7f84641ed360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:20.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.737+0000 7f846c4f7700 1 --2- 172.21.15.78:0/1524599085 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f846406ab40 0x7f84641ed360 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f845400e810 tx=0x7f845400eb20 comp rx=0 tx=0).stop 2023-10-14T11:58:20.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.737+0000 7f846c4f7700 1 -- 172.21.15.78:0/1524599085 shutdown_connections 2023-10-14T11:58:20.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.737+0000 7f846c4f7700 1 --2- 172.21.15.78:0/1524599085 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f84641554e0 0x7f8464150020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.737+0000 7f846c4f7700 1 --2- 172.21.15.78:0/1524599085 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f846406ab40 0x7f84641ed360 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.737+0000 7f846c4f7700 1 --2- 172.21.15.78:0/1524599085 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f844806af00 0x7f844806d3c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.737+0000 7f846c4f7700 1 --2- 172.21.15.78:0/1524599085 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84640c0350 0x7f84640c6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.737+0000 7f846c4f7700 1 -- 172.21.15.78:0/1524599085 >> 172.21.15.78:0/1524599085 conn(0x7f84640c3ac0 msgr2=0x7f8464152760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:20.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.737+0000 7f846c4f7700 1 -- 172.21.15.78:0/1524599085 shutdown_connections 2023-10-14T11:58:20.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.737+0000 7f846c4f7700 1 -- 172.21.15.78:0/1524599085 wait complete. 2023-10-14T11:58:20.752 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-14T11:58:20.752 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-14T11:58:20.753 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-14T11:58:20.753 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-10-14T11:58:20.754 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-10-14T11:58:20.755 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-10-14T11:58:20.755 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-10-14T11:58:20.756 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-10-14T11:58:20.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.964+0000 7face62fa700 1 Processor -- start 2023-10-14T11:58:20.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.964+0000 7face62fa700 1 -- start start 2023-10-14T11:58:20.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.964+0000 7f644f08e700 1 Processor -- start 2023-10-14T11:58:20.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.965+0000 7face62fa700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7face00c0830 0x7face0072a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:20.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.965+0000 7face62fa700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7face00706b0 0x7face01548c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:20.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.965+0000 7face62fa700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face00c18b0 con 0x7face00a7ba0 2023-10-14T11:58:20.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.965+0000 7face62fa700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7face00c15d0 con 0x7face00c0830 2023-10-14T11:58:20.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.965+0000 7face62fa700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7face00c3f30 con 0x7face00706b0 2023-10-14T11:58:20.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.965+0000 7facdffff700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7face00a7ba0 0x7face00c54a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49444/0 (socket says 172.21.15.78:49444) 2023-10-14T11:58:20.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.965+0000 7facdffff700 1 -- 172.21.15.78:0/3970249691 learned_addr learned my addr 172.21.15.78:0/3970249691 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:20.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.965+0000 7facdf7fe700 1 --2- 172.21.15.78:0/3970249691 >> v2:172.21.15.195:3300/0 conn(0x7face00c0830 0x7face0072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:20.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.964+0000 7f644f08e700 1 -- start start 2023-10-14T11:58:20.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.965+0000 7f644f08e700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f64480a7ba0 0x7f6448072a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:20.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.965+0000 7f644f08e700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f64480c0830 0x7f64480c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:20.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.965+0000 7f644f08e700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f64480c59e0 0x7f644806d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:20.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.965+0000 7f644f08e700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64480c18b0 con 0x7f64480c59e0 2023-10-14T11:58:20.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.965+0000 7f644f08e700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64480c15d0 con 0x7f64480a7ba0 2023-10-14T11:58:20.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.965+0000 7f644f08e700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64480c3f30 con 0x7f64480c0830 2023-10-14T11:58:20.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.965+0000 7f6447fff700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f64480c0830 0x7f64480c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:20.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.965+0000 7f644ce2a700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f64480a7ba0 0x7f6448072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:20.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.965+0000 7f6447fff700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f64480c0830 0x7f64480c54a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.78:44242/0 (socket says 172.21.15.78:44242) 2023-10-14T11:58:20.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.965+0000 7f6447fff700 1 -- 172.21.15.78:0/2858077929 learned_addr learned my addr 172.21.15.78:0/2858077929 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:20.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.965+0000 7face4897700 1 --2- 172.21.15.78:0/3970249691 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7face00706b0 0x7face01548c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:20.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.966+0000 7f403af66700 1 Processor -- start 2023-10-14T11:58:20.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.966+0000 7face4897700 1 -- 172.21.15.78:0/3970249691 >> v2:172.21.15.195:3300/0 conn(0x7face00c0830 msgr2=0x7face0072a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:20.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.966+0000 7face4897700 1 --2- 172.21.15.78:0/3970249691 >> v2:172.21.15.195:3300/0 conn(0x7face00c0830 0x7face0072a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.966+0000 7face4897700 1 -- 172.21.15.78:0/3970249691 >> v1:172.21.15.78:6789/0 conn(0x7face00a7ba0 legacy=0x7face00c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:20.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.966+0000 7face4897700 1 -- 172.21.15.78:0/3970249691 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face00c4210 con 0x7face00706b0 2023-10-14T11:58:20.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.966+0000 7f6447fff700 1 -- 172.21.15.78:0/2858077929 >> v2:172.21.15.195:3300/0 conn(0x7f64480a7ba0 msgr2=0x7f6448072a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:20.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.966+0000 7f6447fff700 1 --2- 172.21.15.78:0/2858077929 >> v2:172.21.15.195:3300/0 conn(0x7f64480a7ba0 0x7f6448072a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.966+0000 7f6447fff700 1 -- 172.21.15.78:0/2858077929 >> v2:172.21.15.78:3300/0 conn(0x7f64480c59e0 msgr2=0x7f644806d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:20.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.966+0000 7f6447fff700 1 --2- 172.21.15.78:0/2858077929 >> v2:172.21.15.78:3300/0 conn(0x7f64480c59e0 0x7f644806d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.966+0000 7f6447fff700 1 -- 172.21.15.78:0/2858077929 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64480c4210 con 0x7f64480c0830 2023-10-14T11:58:20.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.966+0000 7face4897700 1 --2- 172.21.15.78:0/3970249691 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7face00706b0 0x7face01548c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7facd8009d90 tx=0x7facd800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=258f3081a136f3c2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:20.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.966+0000 7f6447fff700 1 --2- 172.21.15.78:0/2858077929 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f64480c0830 0x7f64480c54a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f64480a8a20 tx=0x7f643c009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82402c99fd6203aa server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:20.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.966+0000 7facdeffd700 1 -- 172.21.15.78:0/3970249691 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd800e070 con 0x7face00706b0 2023-10-14T11:58:20.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.966+0000 7facdeffd700 1 -- 172.21.15.78:0/3970249691 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd800bb20 con 0x7face00706b0 2023-10-14T11:58:20.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.966+0000 7facdeffd700 1 -- 172.21.15.78:0/3970249691 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd8012430 con 0x7face00706b0 2023-10-14T11:58:20.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.966+0000 7f64477fe700 1 -- 172.21.15.78:0/2858077929 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643c011830 con 0x7f64480c0830 2023-10-14T11:58:20.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.966+0000 7f64477fe700 1 -- 172.21.15.78:0/2858077929 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f643c011990 con 0x7f64480c0830 2023-10-14T11:58:20.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.966+0000 7f64477fe700 1 -- 172.21.15.78:0/2858077929 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643c011d20 con 0x7f64480c0830 2023-10-14T11:58:20.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.967+0000 7f644f08e700 1 -- 172.21.15.78:0/2858077929 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f64480c0830 msgr2=0x7f64480c54a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:20.978 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.967+0000 7f644f08e700 1 --2- 172.21.15.78:0/2858077929 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f64480c0830 0x7f64480c54a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f64480a8a20 tx=0x7f643c009bd0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.978 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.967+0000 7face62fa700 1 -- 172.21.15.78:0/3970249691 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7face00706b0 msgr2=0x7face01548c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:20.979 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.967+0000 7face62fa700 1 --2- 172.21.15.78:0/3970249691 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7face00706b0 0x7face01548c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7facd8009d90 tx=0x7facd800b330 comp rx=0 tx=0).stop 2023-10-14T11:58:20.979 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.968+0000 7face62fa700 1 -- 172.21.15.78:0/3970249691 shutdown_connections 2023-10-14T11:58:20.979 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.968+0000 7face62fa700 1 --2- 172.21.15.78:0/3970249691 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7face00706b0 0x7face01548c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.979 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.968+0000 7face62fa700 1 --2- 172.21.15.78:0/3970249691 >> v2:172.21.15.195:3300/0 conn(0x7face00c0830 0x7face0072a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.979 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.968+0000 7face62fa700 1 -- 172.21.15.78:0/3970249691 >> 172.21.15.78:0/3970249691 conn(0x7face006aab0 msgr2=0x7face006aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:20.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.967+0000 7f403af66700 1 -- start start 2023-10-14T11:58:20.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.968+0000 7f403af66700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f40340a7ba0 0x7f4034072a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:20.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.968+0000 7f403af66700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f40340c0830 0x7f40340c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:20.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.968+0000 7f403af66700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40340c18b0 con 0x7f40340c59e0 2023-10-14T11:58:20.982 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.968+0000 7f403af66700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40340c15d0 con 0x7f40340a7ba0 2023-10-14T11:58:20.982 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.968+0000 7f403af66700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40340c3f30 con 0x7f40340c0830 2023-10-14T11:58:20.982 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.968+0000 7f4038d02700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f40340a7ba0 0x7f4034072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:20.982 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.968+0000 7f4033fff700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f40340c0830 0x7f40340c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:20.982 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.968+0000 7f4038d02700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f40340a7ba0 0x7f4034072a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.78:47316/0 (socket says 172.21.15.78:47316) 2023-10-14T11:58:20.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.968+0000 7f4038d02700 1 -- 172.21.15.78:0/2738120919 learned_addr learned my addr 172.21.15.78:0/2738120919 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:20.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.968+0000 7f4033fff700 1 -- 172.21.15.78:0/2738120919 >> v2:172.21.15.195:3300/0 conn(0x7f40340a7ba0 msgr2=0x7f4034072a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:20.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.968+0000 7f4033fff700 1 --2- 172.21.15.78:0/2738120919 >> v2:172.21.15.195:3300/0 conn(0x7f40340a7ba0 0x7f4034072a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.968+0000 7f4033fff700 1 -- 172.21.15.78:0/2738120919 >> v1:172.21.15.78:6789/0 conn(0x7f40340c59e0 legacy=0x7f403406d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:20.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.968+0000 7f4033fff700 1 -- 172.21.15.78:0/2738120919 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40340c4210 con 0x7f40340c0830 2023-10-14T11:58:20.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.969+0000 7f4038d02700 1 --2- 172.21.15.78:0/2738120919 >> v2:172.21.15.195:3300/0 conn(0x7f40340a7ba0 0x7f4034072a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T11:58:20.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.969+0000 7f4033fff700 1 --2- 172.21.15.78:0/2738120919 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f40340c0830 0x7f40340c54a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4024009840 tx=0x7f4024009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9aa65c0cd533acef server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:20.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.969+0000 7f40337fe700 1 -- 172.21.15.78:0/2738120919 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4024011830 con 0x7f40340c0830 2023-10-14T11:58:20.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.969+0000 7f40337fe700 1 -- 172.21.15.78:0/2738120919 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4024011990 con 0x7f40340c0830 2023-10-14T11:58:20.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.969+0000 7f40337fe700 1 -- 172.21.15.78:0/2738120919 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4024011d20 con 0x7f40340c0830 2023-10-14T11:58:20.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.964+0000 7fdfca888700 1 Processor -- start 2023-10-14T11:58:20.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.964+0000 7fdfca888700 1 -- start start 2023-10-14T11:58:20.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.965+0000 7fdfca888700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdfc40a54d0 0x7fdfc40c0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:20.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.969+0000 7f403af66700 1 -- 172.21.15.78:0/2738120919 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f40340c0830 msgr2=0x7f40340c54a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:20.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.970+0000 7f403af66700 1 --2- 172.21.15.78:0/2738120919 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f40340c0830 0x7f40340c54a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4024009840 tx=0x7f4024009b50 comp rx=0 tx=0).stop 2023-10-14T11:58:20.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.965+0000 7fdfca888700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfc40c4c90 con 0x7fdfc4072500 2023-10-14T11:58:20.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.965+0000 7fdfca888700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfc40c4070 con 0x7fdfc406df90 2023-10-14T11:58:20.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.965+0000 7fdfca888700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfc40c17f0 con 0x7fdfc40a54d0 2023-10-14T11:58:20.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.965+0000 7fdfc3fff700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdfc40a54d0 0x7fdfc40c0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:20.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.965+0000 7fdfc3fff700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdfc40a54d0 0x7fdfc40c0ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44560/0 (socket says 172.21.15.78:44560) 2023-10-14T11:58:20.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.965+0000 7fdfc3fff700 1 -- 172.21.15.78:0/1615667307 learned_addr learned my addr 172.21.15.78:0/1615667307 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:20.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.966+0000 7fdfc3fff700 1 -- 172.21.15.78:0/1615667307 >> v1:172.21.15.195:6789/0 conn(0x7fdfc406df90 legacy=0x7fdfc40c2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:20.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.966+0000 7fdfc3fff700 1 -- 172.21.15.78:0/1615667307 >> v1:172.21.15.78:6789/0 conn(0x7fdfc4072500 legacy=0x7fdfc4072950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:20.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.966+0000 7fdfc3fff700 1 -- 172.21.15.78:0/1615667307 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc40c4350 con 0x7fdfc40a54d0 2023-10-14T11:58:20.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.966+0000 7fdfc3fff700 1 --2- 172.21.15.78:0/1615667307 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfc40a54d0 0x7fdfc40c0ad0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4009870 tx=0x7fdfb4009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac59ecd5d632b5d7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:20.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.967+0000 7fdfc2ffd700 1 -- 172.21.15.78:0/1615667307 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfb4004230 con 0x7fdfc40a54d0 2023-10-14T11:58:20.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.967+0000 7fdfc2ffd700 1 -- 172.21.15.78:0/1615667307 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfb4004390 con 0x7fdfc40a54d0 2023-10-14T11:58:20.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.970+0000 7f403af66700 1 -- 172.21.15.78:0/2738120919 shutdown_connections 2023-10-14T11:58:20.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.970+0000 7f403af66700 1 --2- 172.21.15.78:0/2738120919 >> v2:172.21.15.195:3300/0 conn(0x7f40340a7ba0 0x7f4034072a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.970+0000 7f403af66700 1 --2- 172.21.15.78:0/2738120919 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f40340c0830 0x7f40340c54a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.970+0000 7f403af66700 1 -- 172.21.15.78:0/2738120919 >> 172.21.15.78:0/2738120919 conn(0x7f403406aab0 msgr2=0x7f403406aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:20.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.967+0000 7fdfc2ffd700 1 -- 172.21.15.78:0/1615667307 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfb40036a0 con 0x7fdfc40a54d0 2023-10-14T11:58:20.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.968+0000 7fdfca888700 1 -- 172.21.15.78:0/1615667307 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfc40a54d0 msgr2=0x7fdfc40c0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:20.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.968+0000 7fdfca888700 1 --2- 172.21.15.78:0/1615667307 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfc40a54d0 0x7fdfc40c0ad0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4009870 tx=0x7fdfb4009b80 comp rx=0 tx=0).stop 2023-10-14T11:58:20.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.968+0000 7fdfca888700 1 -- 172.21.15.78:0/1615667307 shutdown_connections 2023-10-14T11:58:20.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.968+0000 7fdfca888700 1 --2- 172.21.15.78:0/1615667307 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfc40a54d0 0x7fdfc40c0ad0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.968+0000 7fdfca888700 1 -- 172.21.15.78:0/1615667307 >> 172.21.15.78:0/1615667307 conn(0x7fdfc406aab0 msgr2=0x7fdfc406aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:20.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.968+0000 7fdfca888700 1 -- 172.21.15.78:0/1615667307 shutdown_connections 2023-10-14T11:58:20.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.968+0000 7fdfca888700 1 -- 172.21.15.78:0/1615667307 wait complete. 2023-10-14T11:58:20.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.970+0000 7face62fa700 1 -- 172.21.15.78:0/3970249691 shutdown_connections 2023-10-14T11:58:20.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.970+0000 7fdfca888700 1 Processor -- start 2023-10-14T11:58:20.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.970+0000 7f403af66700 1 -- 172.21.15.78:0/2738120919 shutdown_connections 2023-10-14T11:58:20.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.970+0000 7f403af66700 1 -- 172.21.15.78:0/2738120919 wait complete. 2023-10-14T11:58:20.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.970+0000 7fdfca888700 1 -- start start 2023-10-14T11:58:20.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.970+0000 7face62fa700 1 -- 172.21.15.78:0/3970249691 wait complete. 2023-10-14T11:58:20.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.971+0000 7face62fa700 1 Processor -- start 2023-10-14T11:58:20.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.971+0000 7fdfca888700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fdfc406df90 0x7fdfc40bace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:20.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.971+0000 7fdfca888700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfc4072500 0x7fdfc40b9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:20.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.971+0000 7fdfca888700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdfc40a54d0 0x7fdfc40b9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:20.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.971+0000 7fdfca888700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc4155fc0 con 0x7fdfc4072500 2023-10-14T11:58:20.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.971+0000 7fdfca888700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc40bcc20 con 0x7fdfc40a54d0 2023-10-14T11:58:20.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.971+0000 7fdfca888700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc4155ce0 con 0x7fdfc406df90 2023-10-14T11:58:20.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.971+0000 7fdfc37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfc4072500 0x7fdfc40b9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:20.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.971+0000 7f403af66700 1 Processor -- start 2023-10-14T11:58:20.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.971+0000 7fdfc37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfc4072500 0x7fdfc40b9aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44586/0 (socket says 172.21.15.78:44586) 2023-10-14T11:58:20.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.972+0000 7fdfc37fe700 1 -- 172.21.15.78:0/1421665583 learned_addr learned my addr 172.21.15.78:0/1421665583 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:20.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.972+0000 7fdfc8e25700 1 --2- 172.21.15.78:0/1421665583 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdfc40a54d0 0x7fdfc40b9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:20.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.972+0000 7face62fa700 1 -- start start 2023-10-14T11:58:20.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.972+0000 7f644f08e700 1 -- 172.21.15.78:0/2858077929 shutdown_connections 2023-10-14T11:58:20.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.972+0000 7f644f08e700 1 --2- 172.21.15.78:0/2858077929 >> v2:172.21.15.78:3300/0 conn(0x7f64480c59e0 0x7f644806d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.972+0000 7f644f08e700 1 --2- 172.21.15.78:0/2858077929 >> v2:172.21.15.195:3300/0 conn(0x7f64480a7ba0 0x7f6448072a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:20.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.972+0000 7f644f08e700 1 --2- 172.21.15.78:0/2858077929 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f64480c0830 0x7f64480c54a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.972+0000 7f644f08e700 1 -- 172.21.15.78:0/2858077929 >> 172.21.15.78:0/2858077929 conn(0x7f644806aab0 msgr2=0x7f644806aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:21.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.970+0000 7f2ebd0b1700 1 Processor -- start 2023-10-14T11:58:21.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.970+0000 7f2ebd0b1700 1 -- start start 2023-10-14T11:58:21.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.971+0000 7f2ebd0b1700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2eb806df90 0x7f2eb80c0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.971+0000 7f2ebd0b1700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2eb8072500 0x7f2eb8072990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.971+0000 7f2ebd0b1700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb80c4c90 con 0x7f2eb80a54d0 2023-10-14T11:58:21.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.971+0000 7f2ebd0b1700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2eb80c17f0 con 0x7f2eb8072500 2023-10-14T11:58:21.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.972+0000 7fcf7a1c2700 1 Processor -- start 2023-10-14T11:58:21.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.971+0000 7f2ebd0b1700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb8072ed0 con 0x7f2eb806df90 2023-10-14T11:58:21.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.972+0000 7f2eb6d9d700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f2eb80a54d0 0x7f2eb80c2f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49466/0 (socket says 172.21.15.78:49466) 2023-10-14T11:58:21.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.972+0000 7f2eb6d9d700 1 -- 172.21.15.78:0/3415871720 learned_addr learned my addr 172.21.15.78:0/3415871720 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:21.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.972+0000 7f2eb759e700 1 --2- 172.21.15.78:0/3415871720 >> v2:172.21.15.78:3300/0 conn(0x7f2eb8072500 0x7f2eb8072990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.972+0000 7fdfc8e25700 1 -- 172.21.15.78:0/1421665583 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fdfc406df90 msgr2=0x7fdfc40bace0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-14T11:58:21.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.972+0000 7fdfc8e25700 1 --2- 172.21.15.78:0/1421665583 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fdfc406df90 0x7fdfc40bace0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.972+0000 7fdfc8e25700 1 -- 172.21.15.78:0/1421665583 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfc4072500 msgr2=0x7fdfc40b9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.972+0000 7fdfc8e25700 1 --2- 172.21.15.78:0/1421665583 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfc4072500 0x7fdfc40b9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.972+0000 7fdfc8e25700 1 -- 172.21.15.78:0/1421665583 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb4009550 con 0x7fdfc40a54d0 2023-10-14T11:58:21.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.972+0000 7fdfc37fe700 1 --2- 172.21.15.78:0/1421665583 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfc4072500 0x7fdfc40b9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T11:58:21.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.972+0000 7fcf7a1c2700 1 -- start start 2023-10-14T11:58:21.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.972+0000 7f2eb759e700 1 -- 172.21.15.78:0/3415871720 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2eb806df90 msgr2=0x7f2eb80c0ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-14T11:58:21.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.972+0000 7f2eb759e700 1 --2- 172.21.15.78:0/3415871720 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2eb806df90 0x7f2eb80c0ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.972+0000 7f2eb759e700 1 -- 172.21.15.78:0/3415871720 >> v1:172.21.15.78:6789/0 conn(0x7f2eb80a54d0 legacy=0x7f2eb80c2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.972+0000 7f2eb759e700 1 -- 172.21.15.78:0/3415871720 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb80c4070 con 0x7f2eb8072500 2023-10-14T11:58:21.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.972+0000 7fdfc8e25700 1 --2- 172.21.15.78:0/1421665583 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdfc40a54d0 0x7fdfc40b9fe0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdfb8001eb0 tx=0x7fdfb800c6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.972+0000 7f2eb759e700 1 --2- 172.21.15.78:0/3415871720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2eb8072500 0x7f2eb8072990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0009d90 tx=0x7f2eb000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f4b567a790a0d70 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7fdfc0ff9700 1 -- 172.21.15.78:0/1421665583 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfb80052d0 con 0x7fdfc40a54d0 2023-10-14T11:58:21.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.972+0000 7face62fa700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7face00706b0 0x7face00a6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7face62fa700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7face00a7ba0 0x7face01e5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7face62fa700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7face00c0830 0x7face01e61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7face62fa700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face01e9200 con 0x7face00706b0 2023-10-14T11:58:21.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7face62fa700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face01e9d80 con 0x7face00c0830 2023-10-14T11:58:21.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7face62fa700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7face01e9450 con 0x7face00a7ba0 2023-10-14T11:58:21.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7facdf7fe700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7face00a7ba0 0x7face01e5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7facdffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7face00706b0 0x7face00a6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7facdf7fe700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7face00a7ba0 0x7face01e5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.78:44266/0 (socket says 172.21.15.78:44266) 2023-10-14T11:58:21.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7facdffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7face00706b0 0x7face00a6a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44600/0 (socket says 172.21.15.78:44600) 2023-10-14T11:58:21.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7facdf7fe700 1 -- 172.21.15.78:0/4166901730 learned_addr learned my addr 172.21.15.78:0/4166901730 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:21.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.972+0000 7f403af66700 1 -- start start 2023-10-14T11:58:21.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7f8ec58dd700 1 Processor -- start 2023-10-14T11:58:21.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7f403af66700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40340a7ba0 0x7f40340bfa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7f403af66700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f40340c0830 0x7f40340bd970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7f403af66700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f40340c59e0 0x7f40340a2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7f403af66700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40340bd550 con 0x7f40340a7ba0 2023-10-14T11:58:21.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7f403af66700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40340a4200 con 0x7f40340c0830 2023-10-14T11:58:21.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7f403af66700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40340a3f20 con 0x7f40340c59e0 2023-10-14T11:58:21.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7fcf7a1c2700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcf740a7ba0 0x7fcf74072a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7fcf7a1c2700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcf740c0830 0x7fcf740c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7fcf7a1c2700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fcf740c59e0 0x7fcf7406d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7fcf7a1c2700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf740c18b0 con 0x7fcf740c0830 2023-10-14T11:58:21.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7fcf7a1c2700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf740c15d0 con 0x7fcf740c59e0 2023-10-14T11:58:21.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7fcf7a1c2700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf740c3f30 con 0x7fcf740a7ba0 2023-10-14T11:58:21.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7f2eb659c700 1 -- 172.21.15.78:0/3415871720 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb000e070 con 0x7f2eb8072500 2023-10-14T11:58:21.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7f2eb659c700 1 -- 172.21.15.78:0/3415871720 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eb000bbd0 con 0x7f2eb8072500 2023-10-14T11:58:21.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7f2eb659c700 1 -- 172.21.15.78:0/3415871720 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb0012430 con 0x7f2eb8072500 2023-10-14T11:58:21.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7f8ec58dd700 1 -- start start 2023-10-14T11:58:21.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7facdffff700 1 -- 172.21.15.78:0/4166901730 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7face00a7ba0 msgr2=0x7face01e5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7facdffff700 1 --2- 172.21.15.78:0/4166901730 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7face00a7ba0 0x7face01e5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7facdffff700 1 -- 172.21.15.78:0/4166901730 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7face00c0830 msgr2=0x7face01e61d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-14T11:58:21.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7facdffff700 1 --2- 172.21.15.78:0/4166901730 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7face00c0830 0x7face01e61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7facdffff700 1 -- 172.21.15.78:0/4166901730 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd8009a70 con 0x7face00706b0 2023-10-14T11:58:21.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7fdfc0ff9700 1 -- 172.21.15.78:0/1421665583 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfb8005450 con 0x7fdfc40a54d0 2023-10-14T11:58:21.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7fdfca888700 1 -- 172.21.15.78:0/1421665583 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfc40ab7f0 con 0x7fdfc40a54d0 2023-10-14T11:58:21.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.973+0000 7fdfca888700 1 -- 172.21.15.78:0/1421665583 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfc40abad0 con 0x7fdfc40a54d0 2023-10-14T11:58:21.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.974+0000 7fdfc0ff9700 1 -- 172.21.15.78:0/1421665583 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfb80038c0 con 0x7fdfc40a54d0 2023-10-14T11:58:21.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.974+0000 7fdfca888700 1 -- 172.21.15.78:0/1421665583 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdfc40c1380 con 0x7fdfc40a54d0 2023-10-14T11:58:21.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.974+0000 7f644f08e700 1 -- 172.21.15.78:0/2858077929 shutdown_connections 2023-10-14T11:58:21.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.974+0000 7fcf737fe700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcf740a7ba0 0x7fcf74072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.974+0000 7fcf737fe700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcf740a7ba0 0x7fcf74072a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.78:44278/0 (socket says 172.21.15.78:44278) 2023-10-14T11:58:21.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.974+0000 7fcf737fe700 1 -- 172.21.15.78:0/760790601 learned_addr learned my addr 172.21.15.78:0/760790601 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:21.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.974+0000 7f4038d02700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40340a7ba0 0x7f40340bfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.974+0000 7f4038d02700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40340a7ba0 0x7f40340bfa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44602/0 (socket says 172.21.15.78:44602) 2023-10-14T11:58:21.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.974+0000 7f4038d02700 1 -- 172.21.15.78:0/1929077980 learned_addr learned my addr 172.21.15.78:0/1929077980 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:21.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.975+0000 7f4033fff700 1 --2- 172.21.15.78:0/1929077980 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f40340c0830 0x7f40340bd970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.975+0000 7f4039503700 1 --2- 172.21.15.78:0/1929077980 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f40340c59e0 0x7f40340a2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.974+0000 7facdffff700 1 --2- 172.21.15.78:0/4166901730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7face00706b0 0x7face00a6a70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7facd400b620 tx=0x7facd400b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.975+0000 7facdcff9700 1 -- 172.21.15.78:0/4166901730 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd400f8d0 con 0x7face00706b0 2023-10-14T11:58:21.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.975+0000 7facdcff9700 1 -- 172.21.15.78:0/4166901730 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd400fa30 con 0x7face00706b0 2023-10-14T11:58:21.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.975+0000 7facdcff9700 1 -- 172.21.15.78:0/4166901730 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd400fdc0 con 0x7face00706b0 2023-10-14T11:58:21.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.974+0000 7fcf72ffd700 1 --2- 172.21.15.78:0/760790601 >> v2:172.21.15.78:3300/0 conn(0x7fcf740c0830 0x7fcf740c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.975+0000 7fcf72ffd700 1 -- 172.21.15.78:0/760790601 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcf740a7ba0 msgr2=0x7fcf74072a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:21.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.975+0000 7fcf72ffd700 1 --2- 172.21.15.78:0/760790601 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcf740a7ba0 0x7fcf74072a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.975+0000 7fcf72ffd700 1 -- 172.21.15.78:0/760790601 >> v2:172.21.15.195:3300/0 conn(0x7fcf740c59e0 msgr2=0x7fcf7406d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:21.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.975+0000 7fcf72ffd700 1 --2- 172.21.15.78:0/760790601 >> v2:172.21.15.195:3300/0 conn(0x7fcf740c59e0 0x7fcf7406d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.975+0000 7fcf72ffd700 1 -- 172.21.15.78:0/760790601 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf740c4210 con 0x7fcf740c0830 2023-10-14T11:58:21.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.975+0000 7fcf72ffd700 1 --2- 172.21.15.78:0/760790601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf740c0830 0x7fcf740c54a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcf740a5480 tx=0x7fcf68009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60d0a2763f7303a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.974+0000 7f8ec58dd700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8ec00a7ba0 0x7f8ec0072a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.974+0000 7f8ec58dd700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec00c18b0 con 0x7f8ec00706b0 2023-10-14T11:58:21.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.974+0000 7f8ec58dd700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec00c15d0 con 0x7f8ec00c0830 2023-10-14T11:58:21.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.974+0000 7f8ec58dd700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ec00c3f30 con 0x7f8ec00a7ba0 2023-10-14T11:58:21.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.975+0000 7f8ebeffd700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8ec00a7ba0 0x7f8ec0072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.975+0000 7f8ebe7fc700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f8ec00c0830 0x7f8ec00c54a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.78:39296/0 (socket says 172.21.15.78:39296) 2023-10-14T11:58:21.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.975+0000 7f8ebe7fc700 1 -- 172.21.15.78:0/2337978563 learned_addr learned my addr 172.21.15.78:0/2337978563 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:21.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.975+0000 7f8ebdffb700 1 -- 172.21.15.78:0/2337978563 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2666463695 0 0) 0x7f8ec00c15d0 con 0x7f8ec00c0830 2023-10-14T11:58:21.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.975+0000 7f8ebdffb700 1 -- 172.21.15.78:0/2337978563 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ea80035b0 con 0x7f8ec00c0830 2023-10-14T11:58:21.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.974+0000 7f2ebd0b1700 1 -- 172.21.15.78:0/3415871720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2eb8072500 msgr2=0x7f2eb8072990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.974+0000 7f2ebd0b1700 1 --2- 172.21.15.78:0/3415871720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2eb8072500 0x7f2eb8072990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0009d90 tx=0x7f2eb000b330 comp rx=0 tx=0).stop 2023-10-14T11:58:21.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.975+0000 7f4810f37700 1 Processor -- start 2023-10-14T11:58:21.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.975+0000 7f4810f37700 1 -- start start 2023-10-14T11:58:21.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.975+0000 7f4039503700 1 -- 172.21.15.78:0/1929077980 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f40340c0830 msgr2=0x7f40340bd970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.975+0000 7f4039503700 1 --2- 172.21.15.78:0/1929077980 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f40340c0830 0x7f40340bd970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.975+0000 7f4039503700 1 -- 172.21.15.78:0/1929077980 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40340a7ba0 msgr2=0x7f40340bfa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.975+0000 7f4039503700 1 --2- 172.21.15.78:0/1929077980 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40340a7ba0 0x7f40340bfa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.975+0000 7f4039503700 1 -- 172.21.15.78:0/1929077980 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f402c009b30 con 0x7f40340c59e0 2023-10-14T11:58:21.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.975+0000 7f8ebdffb700 1 -- 172.21.15.78:0/2337978563 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1721616252 0 0) 0x7f8ec00c18b0 con 0x7f8ec00706b0 2023-10-14T11:58:21.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.975+0000 7f8ebdffb700 1 -- 172.21.15.78:0/2337978563 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ec00c15d0 con 0x7f8ec00706b0 2023-10-14T11:58:21.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.974+0000 7f644f08e700 1 -- 172.21.15.78:0/2858077929 wait complete. 2023-10-14T11:58:21.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.976+0000 7f8ebdffb700 1 -- 172.21.15.78:0/2337978563 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2062395798 0 0) 0x7f8ea80035b0 con 0x7f8ec00c0830 2023-10-14T11:58:21.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.976+0000 7f8ebdffb700 1 -- 172.21.15.78:0/2337978563 >> v2:172.21.15.78:3300/0 conn(0x7f8ec00a7ba0 msgr2=0x7f8ec0072a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:21.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.976+0000 7f8ebdffb700 1 --2- 172.21.15.78:0/2337978563 >> v2:172.21.15.78:3300/0 conn(0x7f8ec00a7ba0 0x7f8ec0072a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.975+0000 7face62fa700 1 -- 172.21.15.78:0/4166901730 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face01e9730 con 0x7face00706b0 2023-10-14T11:58:21.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.975+0000 7face62fa700 1 -- 172.21.15.78:0/4166901730 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face00a6fb0 con 0x7face00706b0 2023-10-14T11:58:21.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.976+0000 7f2ebd0b1700 1 -- 172.21.15.78:0/3415871720 shutdown_connections 2023-10-14T11:58:21.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.976+0000 7f2ebd0b1700 1 --2- 172.21.15.78:0/3415871720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2eb8072500 0x7f2eb8072990 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.976+0000 7f2ebd0b1700 1 --2- 172.21.15.78:0/3415871720 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2eb806df90 0x7f2eb80c0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.976+0000 7f2ebd0b1700 1 -- 172.21.15.78:0/3415871720 >> 172.21.15.78:0/3415871720 conn(0x7f2eb806aab0 msgr2=0x7f2eb806aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:21.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.976+0000 7f8ebdffb700 1 -- 172.21.15.78:0/2337978563 >> v1:172.21.15.78:6789/0 conn(0x7f8ec00706b0 legacy=0x7f8ec01548c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.976+0000 7f8ebdffb700 1 -- 172.21.15.78:0/2337978563 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec00c4210 con 0x7f8ec00c0830 2023-10-14T11:58:21.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.976+0000 7f8ebdffb700 1 -- 172.21.15.78:0/2337978563 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441799917 0 0) 0x7f8eb4000ed0 con 0x7f8ec00c0830 2023-10-14T11:58:21.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.976+0000 7f8ebdffb700 1 -- 172.21.15.78:0/2337978563 >> v1:172.21.15.195:6789/0 conn(0x7f8ec00c0830 legacy=0x7f8ec00c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.976+0000 7f8ebdffb700 1 --2- 172.21.15.78:0/2337978563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ea8003b90 0x7f8ea8003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.976+0000 7f8ebdffb700 1 --2- 172.21.15.78:0/2337978563 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8ea8004850 0x7f8ea8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.976+0000 7f8ebdffb700 1 --2- 172.21.15.78:0/2337978563 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8ea8005210 0x7f8ea8014610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.976+0000 7f8ebdffb700 1 -- 172.21.15.78:0/2337978563 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec00c3f30 con 0x7f8ea8003b90 2023-10-14T11:58:21.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.976+0000 7f8ebdffb700 1 -- 172.21.15.78:0/2337978563 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea8009c10 con 0x7f8ea8005210 2023-10-14T11:58:21.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.976+0000 7f8ebdffb700 1 -- 172.21.15.78:0/2337978563 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea8014b50 con 0x7f8ea8004850 2023-10-14T11:58:21.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.976+0000 7f8ebf7fe700 1 --2- 172.21.15.78:0/2337978563 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8ea8005210 0x7f8ea8014610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.976+0000 7f4038d02700 1 --2- 172.21.15.78:0/1929077980 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40340a7ba0 0x7f40340bfa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T11:58:21.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.976+0000 7f4039503700 1 --2- 172.21.15.78:0/1929077980 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f40340c59e0 0x7f40340a2240 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4028000f60 tx=0x7f402800c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.976+0000 7f4810f37700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f480c0c2b10 0x7f480c0c2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.976+0000 7face62fa700 1 -- 172.21.15.78:0/4166901730 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7face00c11d0 con 0x7face00706b0 2023-10-14T11:58:21.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.976+0000 7f2ebd0b1700 1 -- 172.21.15.78:0/3415871720 shutdown_connections 2023-10-14T11:58:21.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.976+0000 7f4810f37700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f480c072c80 0x7f480c1637a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.976+0000 7f4810f37700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f480c0c5c30 con 0x7f480c0701c0 2023-10-14T11:58:21.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.976+0000 7f4810f37700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f480c06e080 con 0x7f480c0c2b10 2023-10-14T11:58:21.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.976+0000 7f4810f37700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f480c0c5950 con 0x7f480c072c80 2023-10-14T11:58:21.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.976+0000 7f2ebd0b1700 1 -- 172.21.15.78:0/3415871720 wait complete. 2023-10-14T11:58:21.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.976+0000 7f8ebe7fc700 1 --2- 172.21.15.78:0/2337978563 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8ea8004850 0x7f8ea8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.976+0000 7f480ad9d700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f480c072c80 0x7f480c1637a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.977+0000 7f8ebeffd700 1 --2- 172.21.15.78:0/2337978563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ea8003b90 0x7f8ea8003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.977+0000 7f480ad9d700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f480c072c80 0x7f480c1637a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.78:44296/0 (socket says 172.21.15.78:44296) 2023-10-14T11:58:21.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.977+0000 7f480ad9d700 1 -- 172.21.15.78:0/2315680765 learned_addr learned my addr 172.21.15.78:0/2315680765 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:21.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.977+0000 7fcf727fc700 1 -- 172.21.15.78:0/760790601 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf68010a90 con 0x7fcf740c0830 2023-10-14T11:58:21.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.976+0000 7f644f08e700 1 Processor -- start 2023-10-14T11:58:21.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.977+0000 7f480ad9d700 1 -- 172.21.15.78:0/2315680765 >> v2:172.21.15.78:3300/0 conn(0x7f480c0c2b10 msgr2=0x7f480c0c2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:21.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.977+0000 7f480ad9d700 1 --2- 172.21.15.78:0/2315680765 >> v2:172.21.15.78:3300/0 conn(0x7f480c0c2b10 0x7f480c0c2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.977+0000 7f480ad9d700 1 -- 172.21.15.78:0/2315680765 >> v1:172.21.15.195:6789/0 conn(0x7f480c0701c0 legacy=0x7f480c072570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:21.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.977+0000 7f480ad9d700 1 -- 172.21.15.78:0/2315680765 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f480c0c0d20 con 0x7f480c072c80 2023-10-14T11:58:21.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.977+0000 7f480ad9d700 1 --2- 172.21.15.78:0/2315680765 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f480c072c80 0x7f480c1637a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4804009e50 tx=0x7f480400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b65385437158d839 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.977+0000 7fdfc0ff9700 1 -- 172.21.15.78:0/1421665583 <== mon.1 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fdfb8003a20 con 0x7fdfc40a54d0 2023-10-14T11:58:21.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.977+0000 7fdfc0ff9700 1 --2- 172.21.15.78:0/1421665583 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fdfb006afd0 0x7fdfb006d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.977+0000 7fdfc0ff9700 1 -- 172.21.15.78:0/1421665583 <== mon.1 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdfb8088fe0 con 0x7fdfc40a54d0 2023-10-14T11:58:21.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.978+0000 7fdfc0ff9700 1 --2- 172.21.15.78:0/1421665583 >> [v2:172.21.15.78:6808/2176928406,v1:172.21.15.78:6809/2176928406] conn(0x7fdfb00731f0 0x7fdfb0075610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.978+0000 7fdfc0ff9700 1 -- 172.21.15.78:0/1421665583 --> [v2:172.21.15.78:6808/2176928406,v1:172.21.15.78:6809/2176928406] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdfb0075cc0 con 0x7fdfb00731f0 2023-10-14T11:58:21.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.978+0000 7fdfc0ff9700 1 -- 172.21.15.78:0/1421665583 <== mon.1 v2:172.21.15.195:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdfb8089390 con 0x7fdfc40a54d0 2023-10-14T11:58:21.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.978+0000 7f8ebeffd700 1 -- 172.21.15.78:0/2337978563 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8ea8004850 msgr2=0x7f8ea8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:21.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.978+0000 7f8ebeffd700 1 --2- 172.21.15.78:0/2337978563 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8ea8004850 0x7f8ea8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.978+0000 7f8ebeffd700 1 -- 172.21.15.78:0/2337978563 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8ea8005210 msgr2=0x7f8ea8014610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:21.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.978+0000 7f8ebeffd700 1 --2- 172.21.15.78:0/2337978563 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8ea8005210 0x7f8ea8014610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.978+0000 7f8ebeffd700 1 -- 172.21.15.78:0/2337978563 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec00c4210 con 0x7f8ea8003b90 2023-10-14T11:58:21.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.977+0000 7fcf727fc700 1 -- 172.21.15.78:0/760790601 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf68010bf0 con 0x7fcf740c0830 2023-10-14T11:58:21.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.977+0000 7fcf727fc700 1 -- 172.21.15.78:0/760790601 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf68018440 con 0x7fcf740c0830 2023-10-14T11:58:21.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.978+0000 7fcf7a1c2700 1 -- 172.21.15.78:0/760790601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf740c0830 msgr2=0x7fcf740c54a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.978+0000 7fcf7a1c2700 1 --2- 172.21.15.78:0/760790601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf740c0830 0x7fcf740c54a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcf740a5480 tx=0x7fcf68009b20 comp rx=0 tx=0).stop 2023-10-14T11:58:21.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.978+0000 7fdfc37fe700 1 --2- 172.21.15.78:0/1421665583 >> [v2:172.21.15.78:6808/2176928406,v1:172.21.15.78:6809/2176928406] conn(0x7fdfb00731f0 0x7fdfb0075610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.978+0000 7f8ebeffd700 1 --2- 172.21.15.78:0/2337978563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ea8003b90 0x7f8ea8003f90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8eb000bc80 tx=0x7f8ea8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3b0d2770cd3149e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.978+0000 7facdcff9700 1 -- 172.21.15.78:0/4166901730 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7facd400a070 con 0x7face00706b0 2023-10-14T11:58:21.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.977+0000 7f40317fa700 1 -- 172.21.15.78:0/1929077980 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40280052d0 con 0x7f40340c59e0 2023-10-14T11:58:21.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.977+0000 7f40317fa700 1 -- 172.21.15.78:0/1929077980 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4028005430 con 0x7f40340c59e0 2023-10-14T11:58:21.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.977+0000 7f40317fa700 1 -- 172.21.15.78:0/1929077980 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40280038c0 con 0x7f40340c59e0 2023-10-14T11:58:21.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.977+0000 7f403af66700 1 -- 172.21.15.78:0/1929077980 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40240094a0 con 0x7f40340c59e0 2023-10-14T11:58:21.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.977+0000 7f403af66700 1 -- 172.21.15.78:0/1929077980 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40340bd270 con 0x7f40340c59e0 2023-10-14T11:58:21.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.977+0000 7f480959a700 1 -- 172.21.15.78:0/2315680765 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f480400e070 con 0x7f480c072c80 2023-10-14T11:58:21.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.977+0000 7f480959a700 1 -- 172.21.15.78:0/2315680765 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4804003a20 con 0x7f480c072c80 2023-10-14T11:58:21.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.977+0000 7f480959a700 1 -- 172.21.15.78:0/2315680765 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4804003dd0 con 0x7f480c072c80 2023-10-14T11:58:21.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.978+0000 7f4810f37700 1 -- 172.21.15.78:0/2315680765 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f480c072c80 msgr2=0x7f480c1637a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.978+0000 7f4810f37700 1 --2- 172.21.15.78:0/2315680765 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f480c072c80 0x7f480c1637a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4804009e50 tx=0x7f480400b330 comp rx=0 tx=0).stop 2023-10-14T11:58:21.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.979+0000 7fdfc37fe700 1 --2- 172.21.15.78:0/1421665583 >> [v2:172.21.15.78:6808/2176928406,v1:172.21.15.78:6809/2176928406] conn(0x7fdfb00731f0 0x7fdfb0075610 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.979+0000 7fcf7a1c2700 1 -- 172.21.15.78:0/760790601 shutdown_connections 2023-10-14T11:58:21.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.979+0000 7fcf7a1c2700 1 --2- 172.21.15.78:0/760790601 >> v2:172.21.15.195:3300/0 conn(0x7fcf740c59e0 0x7fcf7406d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.979+0000 7fcf7a1c2700 1 --2- 172.21.15.78:0/760790601 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcf740a7ba0 0x7fcf74072a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.979+0000 7fcf7a1c2700 1 --2- 172.21.15.78:0/760790601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf740c0830 0x7fcf740c54a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.979+0000 7fcf7a1c2700 1 -- 172.21.15.78:0/760790601 >> 172.21.15.78:0/760790601 conn(0x7fcf7406aab0 msgr2=0x7fcf7406aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:21.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.979+0000 7f8ebf7fe700 1 -- 172.21.15.78:0/2337978563 reap_dead start 2023-10-14T11:58:21.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.979+0000 7f8ebdffb700 1 -- 172.21.15.78:0/2337978563 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb0004220 con 0x7f8ea8003b90 2023-10-14T11:58:21.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.979+0000 7f8ebdffb700 1 -- 172.21.15.78:0/2337978563 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb0004380 con 0x7f8ea8003b90 2023-10-14T11:58:21.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.979+0000 7f8ebdffb700 1 -- 172.21.15.78:0/2337978563 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb000e910 con 0x7f8ea8003b90 2023-10-14T11:58:21.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.979+0000 7fcf7a1c2700 1 -- 172.21.15.78:0/760790601 shutdown_connections 2023-10-14T11:58:21.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.979+0000 7fdfc3fff700 1 --2- 172.21.15.78:0/1421665583 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fdfb006afd0 0x7fdfb006d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.979+0000 7f403af66700 1 -- 172.21.15.78:0/1929077980 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4014000fc0 con 0x7f40340c59e0 2023-10-14T11:58:21.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.978+0000 7f2ebd0b1700 1 Processor -- start 2023-10-14T11:58:21.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.978+0000 7f644f08e700 1 -- start start 2023-10-14T11:58:21.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.979+0000 7f644f08e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64480a7ba0 0x7f64481ee2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.979+0000 7f644f08e700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f64480c0830 0x7f64480bdec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.979+0000 7f644f08e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f64480c59e0 0x7f64481f6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.979+0000 7f644f08e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64480bdaa0 con 0x7f64480a7ba0 2023-10-14T11:58:21.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.979+0000 7f644f08e700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64481f7290 con 0x7f64480c59e0 2023-10-14T11:58:21.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.979+0000 7f644f08e700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64481ee7f0 con 0x7f64480c0830 2023-10-14T11:58:21.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.979+0000 7facdcff9700 1 --2- 172.21.15.78:0/4166901730 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7faccc06af00 0x7faccc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.979+0000 7facdcff9700 1 -- 172.21.15.78:0/4166901730 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7facd408a280 con 0x7face00706b0 2023-10-14T11:58:21.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.979+0000 7facdcff9700 1 --2- 172.21.15.78:0/4166901730 >> [v2:172.21.15.78:6816/3584112288,v1:172.21.15.78:6817/3584112288] conn(0x7faccc073120 0x7faccc075540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.979+0000 7facdcff9700 1 -- 172.21.15.78:0/4166901730 --> [v2:172.21.15.78:6816/3584112288,v1:172.21.15.78:6817/3584112288] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faccc075bf0 con 0x7faccc073120 2023-10-14T11:58:21.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.979+0000 7facdcff9700 1 -- 172.21.15.78:0/4166901730 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7facd408a630 con 0x7face00706b0 2023-10-14T11:58:21.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.979+0000 7facdf7fe700 1 --2- 172.21.15.78:0/4166901730 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7faccc06af00 0x7faccc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.979+0000 7face4897700 1 --2- 172.21.15.78:0/4166901730 >> [v2:172.21.15.78:6816/3584112288,v1:172.21.15.78:6817/3584112288] conn(0x7faccc073120 0x7faccc075540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.980+0000 7f8ec58dd700 1 -- 172.21.15.78:0/2337978563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ea8003b90 msgr2=0x7f8ea8003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.980+0000 7f8ec58dd700 1 --2- 172.21.15.78:0/2337978563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ea8003b90 0x7f8ea8003f90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8eb000bc80 tx=0x7f8ea8005880 comp rx=0 tx=0).stop 2023-10-14T11:58:21.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.980+0000 7f644d62b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f64480c59e0 0x7f64481f6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.980+0000 7f644d62b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f64480c59e0 0x7f64481f6450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.78:47384/0 (socket says 172.21.15.78:47384) 2023-10-14T11:58:21.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.980+0000 7f644d62b700 1 -- 172.21.15.78:0/2338394425 learned_addr learned my addr 172.21.15.78:0/2338394425 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:21.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.980+0000 7f6447fff700 1 --2- 172.21.15.78:0/2338394425 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f64480c0830 0x7f64480bdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.980+0000 7f644ce2a700 1 --2- 172.21.15.78:0/2338394425 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64480a7ba0 0x7f64481ee2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.980+0000 7f6447fff700 1 -- 172.21.15.78:0/2338394425 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f64480c59e0 msgr2=0x7f64481f6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.980+0000 7f6447fff700 1 --2- 172.21.15.78:0/2338394425 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f64480c59e0 0x7f64481f6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.980+0000 7f6447fff700 1 -- 172.21.15.78:0/2338394425 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64480a7ba0 msgr2=0x7f64481ee2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.980+0000 7f6447fff700 1 --2- 172.21.15.78:0/2338394425 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64480a7ba0 0x7f64481ee2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.980+0000 7f6447fff700 1 -- 172.21.15.78:0/2338394425 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f643c009550 con 0x7f64480c0830 2023-10-14T11:58:21.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.980+0000 7f644ce2a700 1 --2- 172.21.15.78:0/2338394425 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64480a7ba0 0x7f64481ee2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:21.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.980+0000 7fdfc0ff9700 1 -- 172.21.15.78:0/1421665583 <== osd.1 v2:172.21.15.78:6808/2176928406 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fdfb0075cc0 con 0x7fdfb00731f0 2023-10-14T11:58:21.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.980+0000 7fdfc3fff700 1 --2- 172.21.15.78:0/1421665583 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fdfb006afd0 0x7fdfb006d490 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4009bc0 tx=0x7fdfb4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.980+0000 7face4897700 1 --2- 172.21.15.78:0/4166901730 >> [v2:172.21.15.78:6816/3584112288,v1:172.21.15.78:6817/3584112288] conn(0x7faccc073120 0x7faccc075540 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.979+0000 7fcf7a1c2700 1 -- 172.21.15.78:0/760790601 wait complete. 2023-10-14T11:58:21.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.980+0000 7f8ec58dd700 1 -- 172.21.15.78:0/2337978563 shutdown_connections 2023-10-14T11:58:21.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.980+0000 7f8ec58dd700 1 --2- 172.21.15.78:0/2337978563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ea8003b90 0x7f8ea8003f90 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.980+0000 7f8ec58dd700 1 -- 172.21.15.78:0/2337978563 >> 172.21.15.78:0/2337978563 conn(0x7f8ec006aab0 msgr2=0x7f8ec006aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:21.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.980+0000 7f8ec58dd700 1 -- 172.21.15.78:0/2337978563 shutdown_connections 2023-10-14T11:58:21.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.980+0000 7f8ec58dd700 1 -- 172.21.15.78:0/2337978563 wait complete. 2023-10-14T11:58:21.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.980+0000 7fcf7a1c2700 1 Processor -- start 2023-10-14T11:58:21.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.980+0000 7f6447fff700 1 --2- 172.21.15.78:0/2338394425 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f64480c0830 0x7f64480bdec0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f643c016000 tx=0x7f643c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.981+0000 7f64457fa700 1 -- 172.21.15.78:0/2338394425 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643c004510 con 0x7f64480c0830 2023-10-14T11:58:21.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.981+0000 7f64457fa700 1 -- 172.21.15.78:0/2338394425 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f643c0036a0 con 0x7f64480c0830 2023-10-14T11:58:21.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.981+0000 7f64457fa700 1 -- 172.21.15.78:0/2338394425 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643c003a50 con 0x7f64480c0830 2023-10-14T11:58:21.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.981+0000 7f644f08e700 1 -- 172.21.15.78:0/2338394425 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64480a4ef0 con 0x7f64480c0830 2023-10-14T11:58:21.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.981+0000 7f644f08e700 1 -- 172.21.15.78:0/2338394425 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64480bf830 con 0x7f64480c0830 2023-10-14T11:58:21.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.981+0000 7fcf7a1c2700 1 -- start start 2023-10-14T11:58:21.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.982+0000 7fcf7a1c2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf740a7ba0 0x7fcf741fa830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.982+0000 7fcf7a1c2700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcf740c0830 0x7fcf7414d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.982+0000 7fcf7a1c2700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcf740c59e0 0x7fcf741ec830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.982+0000 7fcf7a1c2700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf741fad70 con 0x7fcf740a7ba0 2023-10-14T11:58:21.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.982+0000 7fcf7a1c2700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf740b94c0 con 0x7fcf740c0830 2023-10-14T11:58:21.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.982+0000 7fcf7a1c2700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf740a8dd0 con 0x7fcf740c59e0 2023-10-14T11:58:21.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.979+0000 7f2ebd0b1700 1 -- start start 2023-10-14T11:58:21.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.980+0000 7f2ebd0b1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2eb806df90 0x7f2eb8154710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.980+0000 7f2ebd0b1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2eb8072500 0x7f2eb80bf800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.980+0000 7f2ebd0b1700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2eb80a54d0 0x7f2eb814d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.980+0000 7f2ebd0b1700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb8154c50 con 0x7f2eb8072500 2023-10-14T11:58:21.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.980+0000 7f2ebd0b1700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb80b9d90 con 0x7f2eb806df90 2023-10-14T11:58:21.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.980+0000 7f2ebd0b1700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb80a64c0 con 0x7f2eb80a54d0 2023-10-14T11:58:21.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.981+0000 7f2eaffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2eb8072500 0x7f2eb80bf800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.981+0000 7f2eaffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2eb8072500 0x7f2eb80bf800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44650/0 (socket says 172.21.15.78:44650) 2023-10-14T11:58:21.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.981+0000 7f2eaffff700 1 -- 172.21.15.78:0/1074253911 learned_addr learned my addr 172.21.15.78:0/1074253911 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:21.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.981+0000 7f2eaffff700 1 -- 172.21.15.78:0/1074253911 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2eb80a54d0 msgr2=0x7f2eb814d670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T11:58:21.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.981+0000 7f2eaffff700 1 --2- 172.21.15.78:0/1074253911 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2eb80a54d0 0x7f2eb814d670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.981+0000 7f2eaffff700 1 -- 172.21.15.78:0/1074253911 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2eb806df90 msgr2=0x7f2eb8154710 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-14T11:58:21.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.981+0000 7f2eaffff700 1 --2- 172.21.15.78:0/1074253911 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2eb806df90 0x7f2eb8154710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.981+0000 7f2eaffff700 1 -- 172.21.15.78:0/1074253911 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb0009a70 con 0x7f2eb8072500 2023-10-14T11:58:21.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.982+0000 7f2eaffff700 1 --2- 172.21.15.78:0/1074253911 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2eb8072500 0x7f2eb80bf800 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2ea000b550 tx=0x7f2ea000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.069 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.982+0000 7fcf72ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcf740c0830 0x7fcf7414d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.069 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.982+0000 7fcf72ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcf740c0830 0x7fcf7414d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.78:47394/0 (socket says 172.21.15.78:47394) 2023-10-14T11:58:21.069 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.982+0000 7fcf72ffd700 1 -- 172.21.15.78:0/453434029 learned_addr learned my addr 172.21.15.78:0/453434029 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:21.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.982+0000 7f4810f37700 1 -- 172.21.15.78:0/2315680765 shutdown_connections 2023-10-14T11:58:21.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.982+0000 7f4810f37700 1 --2- 172.21.15.78:0/2315680765 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f480c072c80 0x7f480c1637a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.982+0000 7f4810f37700 1 --2- 172.21.15.78:0/2315680765 >> v2:172.21.15.78:3300/0 conn(0x7f480c0c2b10 0x7f480c0c2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.982+0000 7f4810f37700 1 -- 172.21.15.78:0/2315680765 >> 172.21.15.78:0/2315680765 conn(0x7f480c06aab0 msgr2=0x7f480c06aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:21.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.982+0000 7f2eaf7fe700 1 -- 172.21.15.78:0/1074253911 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea0011830 con 0x7f2eb8072500 2023-10-14T11:58:21.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.982+0000 7f2ebd0b1700 1 -- 172.21.15.78:0/1074253911 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eb80ba880 con 0x7f2eb8072500 2023-10-14T11:58:21.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.982+0000 7f8ec58dd700 1 Processor -- start 2023-10-14T11:58:21.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.982+0000 7f2ebd0b1700 1 -- 172.21.15.78:0/1074253911 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eb80ba2c0 con 0x7f2eb8072500 2023-10-14T11:58:21.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.982+0000 7f40317fa700 1 -- 172.21.15.78:0/1929077980 <== mon.2 v2:172.21.15.195:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4028003a20 con 0x7f40340c59e0 2023-10-14T11:58:21.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.982+0000 7f40317fa700 1 --2- 172.21.15.78:0/1929077980 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f402006af50 0x7f402006d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.982+0000 7f40317fa700 1 -- 172.21.15.78:0/1929077980 <== mon.2 v2:172.21.15.195:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4028088f10 con 0x7f40340c59e0 2023-10-14T11:58:21.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.982+0000 7f40317fa700 1 --2- 172.21.15.78:0/1929077980 >> [v2:172.21.15.78:6824/3647044577,v1:172.21.15.78:6825/3647044577] conn(0x7f4020073170 0x7f4020075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.982+0000 7f40317fa700 1 -- 172.21.15.78:0/1929077980 --> [v2:172.21.15.78:6824/3647044577,v1:172.21.15.78:6825/3647044577] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4020075c40 con 0x7f4020073170 2023-10-14T11:58:21.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.982+0000 7f40317fa700 1 -- 172.21.15.78:0/1929077980 <== mon.2 v2:172.21.15.195:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4028089280 con 0x7f40340c59e0 2023-10-14T11:58:21.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.983+0000 7facdcff9700 1 -- 172.21.15.78:0/4166901730 <== osd.3 v2:172.21.15.78:6816/3584112288 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7faccc075bf0 con 0x7faccc073120 2023-10-14T11:58:21.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.982+0000 7fcf73fff700 1 --2- 172.21.15.78:0/453434029 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcf740c59e0 0x7fcf741ec830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.983+0000 7fcf72ffd700 1 -- 172.21.15.78:0/453434029 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcf740c59e0 msgr2=0x7fcf741ec830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.983+0000 7fcf72ffd700 1 --2- 172.21.15.78:0/453434029 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcf740c59e0 0x7fcf741ec830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.983+0000 7fcf72ffd700 1 -- 172.21.15.78:0/453434029 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf740a7ba0 msgr2=0x7fcf741fa830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.983+0000 7fcf72ffd700 1 --2- 172.21.15.78:0/453434029 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf740a7ba0 0x7fcf741fa830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.983+0000 7fcf72ffd700 1 -- 172.21.15.78:0/453434029 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf680094a0 con 0x7fcf740c0830 2023-10-14T11:58:21.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.983+0000 7f4810f37700 1 -- 172.21.15.78:0/2315680765 shutdown_connections 2023-10-14T11:58:21.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.983+0000 7f644f08e700 1 -- 172.21.15.78:0/2338394425 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f64480a7f80 con 0x7f64480c0830 2023-10-14T11:58:21.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.983+0000 7f64457fa700 1 -- 172.21.15.78:0/2338394425 <== mon.2 v2:172.21.15.195:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f643c003bb0 con 0x7f64480c0830 2023-10-14T11:58:21.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.984+0000 7f64457fa700 1 --2- 172.21.15.78:0/2338394425 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f642c06aed0 0x7f642c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.984+0000 7f64457fa700 1 -- 172.21.15.78:0/2338394425 <== mon.2 v2:172.21.15.195:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f643c08a7d0 con 0x7f64480c0830 2023-10-14T11:58:21.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.984+0000 7f64457fa700 1 --2- 172.21.15.78:0/2338394425 >> [v2:172.21.15.78:6800/3884969780,v1:172.21.15.78:6801/3884969780] conn(0x7f642c073070 0x7f642c075490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.984+0000 7f64457fa700 1 -- 172.21.15.78:0/2338394425 --> [v2:172.21.15.78:6800/3884969780,v1:172.21.15.78:6801/3884969780] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f642c075b40 con 0x7f642c073070 2023-10-14T11:58:21.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.984+0000 7f64457fa700 1 -- 172.21.15.78:0/2338394425 <== mon.2 v2:172.21.15.195:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f643c006ce0 con 0x7f64480c0830 2023-10-14T11:58:21.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.985+0000 7f644d62b700 1 --2- 172.21.15.78:0/2338394425 >> [v2:172.21.15.78:6800/3884969780,v1:172.21.15.78:6801/3884969780] conn(0x7f642c073070 0x7f642c075490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.983+0000 7f4038d02700 1 --2- 172.21.15.78:0/1929077980 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f402006af50 0x7f402006d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.985+0000 7f4033fff700 1 --2- 172.21.15.78:0/1929077980 >> [v2:172.21.15.78:6824/3647044577,v1:172.21.15.78:6825/3647044577] conn(0x7f4020073170 0x7f4020075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.985+0000 7facdf7fe700 1 --2- 172.21.15.78:0/4166901730 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7faccc06af00 0x7faccc06d3c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7face0072540 tx=0x7facd000a320 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.982+0000 7f2eaf7fe700 1 -- 172.21.15.78:0/1074253911 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea0011990 con 0x7f2eb8072500 2023-10-14T11:58:21.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.982+0000 7f2eaf7fe700 1 -- 172.21.15.78:0/1074253911 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea0011d40 con 0x7f2eb8072500 2023-10-14T11:58:21.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.984+0000 7f2ebd0b1700 1 -- 172.21.15.78:0/1074253911 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2e9c000fc0 con 0x7f2eb8072500 2023-10-14T11:58:21.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.985+0000 7f2eaf7fe700 1 -- 172.21.15.78:0/1074253911 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2ea001a400 con 0x7f2eb8072500 2023-10-14T11:58:21.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.983+0000 7f4810f37700 1 -- 172.21.15.78:0/2315680765 wait complete. 2023-10-14T11:58:21.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.985+0000 7f644d62b700 1 --2- 172.21.15.78:0/2338394425 >> [v2:172.21.15.78:6800/3884969780,v1:172.21.15.78:6801/3884969780] conn(0x7f642c073070 0x7f642c075490 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.986+0000 7f644ce2a700 1 --2- 172.21.15.78:0/2338394425 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f642c06aed0 0x7f642c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.983+0000 7f8ec58dd700 1 -- start start 2023-10-14T11:58:21.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.984+0000 7f8ec58dd700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8ea8003b90 0x7f8ec00a5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.984+0000 7f8ec58dd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8ec00a67d0 0x7f8ec01ecfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.984+0000 7f8ec58dd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ec01ed500 0x7f8ec00bf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.984+0000 7f8ec58dd700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec00bdee0 con 0x7f8ec01ed500 2023-10-14T11:58:21.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.984+0000 7f8ec58dd700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec00ab390 con 0x7f8ec00a67d0 2023-10-14T11:58:21.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.984+0000 7f8ec58dd700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec01edd70 con 0x7f8ea8003b90 2023-10-14T11:58:21.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.985+0000 7f8ebeffd700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8ea8003b90 0x7f8ec00a5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.985+0000 7f8ebeffd700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8ea8003b90 0x7f8ec00a5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.78:44336/0 (socket says 172.21.15.78:44336) 2023-10-14T11:58:21.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.985+0000 7f8ebeffd700 1 -- 172.21.15.78:0/196405721 learned_addr learned my addr 172.21.15.78:0/196405721 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:21.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.986+0000 7f8ebe7fc700 1 --2- 172.21.15.78:0/196405721 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8ec00a67d0 0x7f8ec01ecfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.985+0000 7f2eaf7fe700 1 --2- 172.21.15.78:0/1074253911 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f2e9806af00 0x7f2e9806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.985+0000 7f2eaf7fe700 1 -- 172.21.15.78:0/1074253911 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ea0089590 con 0x7f2eb8072500 2023-10-14T11:58:21.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.985+0000 7f2eaf7fe700 1 --2- 172.21.15.78:0/1074253911 >> [v2:172.21.15.195:6808/72554374,v1:172.21.15.195:6810/72554374] conn(0x7f2e980730a0 0x7f2e980754c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.985+0000 7f2eaf7fe700 1 -- 172.21.15.78:0/1074253911 --> [v2:172.21.15.195:6808/72554374,v1:172.21.15.195:6810/72554374] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2e98075b70 con 0x7f2e980730a0 2023-10-14T11:58:21.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.985+0000 7f2eaf7fe700 1 -- 172.21.15.78:0/1074253911 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ea008bc30 con 0x7f2eb8072500 2023-10-14T11:58:21.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.986+0000 7f2eb6d9d700 1 --2- 172.21.15.78:0/1074253911 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f2e9806af00 0x7f2e9806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.985+0000 7f4810f37700 1 Processor -- start 2023-10-14T11:58:21.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.985+0000 7f4810f37700 1 -- start start 2023-10-14T11:58:21.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.986+0000 7f644ce2a700 1 --2- 172.21.15.78:0/2338394425 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f642c06aed0 0x7f642c06d390 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f64480719a0 tx=0x7f6438008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.086 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.986+0000 7f4033fff700 1 --2- 172.21.15.78:0/1929077980 >> [v2:172.21.15.78:6824/3647044577,v1:172.21.15.78:6825/3647044577] conn(0x7f4020073170 0x7f4020075590 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.086 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.985+0000 7fcf72ffd700 1 --2- 172.21.15.78:0/453434029 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcf740c0830 0x7fcf7414d7e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcf68009b60 tx=0x7fcf68004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.086 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.986+0000 7f8ebe7fc700 1 -- 172.21.15.78:0/196405721 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8ea8003b90 msgr2=0x7f8ec00a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.087 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.986+0000 7f8ebe7fc700 1 --2- 172.21.15.78:0/196405721 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8ea8003b90 0x7f8ec00a5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.087 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.986+0000 7f8ebe7fc700 1 -- 172.21.15.78:0/196405721 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ec01ed500 msgr2=0x7f8ec00bf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.087 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.986+0000 7f8ebe7fc700 1 --2- 172.21.15.78:0/196405721 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ec01ed500 0x7f8ec00bf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.087 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.986+0000 7f8ebe7fc700 1 -- 172.21.15.78:0/196405721 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec00ab5a0 con 0x7f8ec00a67d0 2023-10-14T11:58:21.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.986+0000 7f8ebe7fc700 1 --2- 172.21.15.78:0/196405721 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8ec00a67d0 0x7f8ec01ecfc0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4010d60 tx=0x7f8eb400e620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.986+0000 7fcf5bfff700 1 -- 172.21.15.78:0/453434029 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf68003c20 con 0x7fcf740c0830 2023-10-14T11:58:21.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.986+0000 7fcf5bfff700 1 -- 172.21.15.78:0/453434029 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf68003da0 con 0x7fcf740c0830 2023-10-14T11:58:21.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.986+0000 7fcf5bfff700 1 -- 172.21.15.78:0/453434029 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf68018680 con 0x7fcf740c0830 2023-10-14T11:58:21.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.988+0000 7f64457fa700 1 -- 172.21.15.78:0/2338394425 <== osd.0 v2:172.21.15.78:6800/3884969780 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f642c075b40 con 0x7f642c073070 2023-10-14T11:58:21.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.986+0000 7f4038d02700 1 --2- 172.21.15.78:0/1929077980 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f402006af50 0x7f402006d410 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f402c009fd0 tx=0x7f402c00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.987+0000 7f40317fa700 1 -- 172.21.15.78:0/1929077980 <== osd.2 v2:172.21.15.78:6824/3647044577 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f4020075c40 con 0x7f4020073170 2023-10-14T11:58:21.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.987+0000 7fcf7a1c2700 1 -- 172.21.15.78:0/453434029 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf740a90b0 con 0x7fcf740c0830 2023-10-14T11:58:21.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.987+0000 7fcf7a1c2700 1 -- 172.21.15.78:0/453434029 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf740be990 con 0x7fcf740c0830 2023-10-14T11:58:21.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.988+0000 7fcf7a1c2700 1 -- 172.21.15.78:0/453434029 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcf740a7f80 con 0x7fcf740c0830 2023-10-14T11:58:21.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.989+0000 7fcf5bfff700 1 -- 172.21.15.78:0/453434029 <== mon.1 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fcf680187e0 con 0x7fcf740c0830 2023-10-14T11:58:21.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.988+0000 7f8ea7fff700 1 -- 172.21.15.78:0/196405721 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb400ee30 con 0x7f8ec00a67d0 2023-10-14T11:58:21.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.988+0000 7f8ea7fff700 1 -- 172.21.15.78:0/196405721 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb4015070 con 0x7f8ec00a67d0 2023-10-14T11:58:21.092 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.988+0000 7f8ea7fff700 1 -- 172.21.15.78:0/196405721 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb4011670 con 0x7f8ec00a67d0 2023-10-14T11:58:21.092 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.988+0000 7f8ec58dd700 1 -- 172.21.15.78:0/196405721 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ec01ed980 con 0x7f8ec00a67d0 2023-10-14T11:58:21.092 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.988+0000 7f8ec58dd700 1 -- 172.21.15.78:0/196405721 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ec00ab880 con 0x7f8ec00a67d0 2023-10-14T11:58:21.092 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.987+0000 7f2eb6d9d700 1 --2- 172.21.15.78:0/1074253911 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f2e9806af00 0x7f2e9806d3c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8151ab0 tx=0x7f2ea8007ad0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.988+0000 7f2eb759e700 1 --2- 172.21.15.78:0/1074253911 >> [v2:172.21.15.195:6808/72554374,v1:172.21.15.195:6810/72554374] conn(0x7f2e980730a0 0x7f2e980754c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.988+0000 7f2eb759e700 1 --2- 172.21.15.78:0/1074253911 >> [v2:172.21.15.195:6808/72554374,v1:172.21.15.195:6810/72554374] conn(0x7f2e980730a0 0x7f2e980754c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.986+0000 7f4810f37700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f480c0701c0 0x7f480c1f1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.986+0000 7f4810f37700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f480c072c80 0x7f480c1ee600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.986+0000 7f4810f37700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f480c0c2b10 0x7f480c0be450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.986+0000 7f4810f37700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f480c1ee1e0 con 0x7f480c0701c0 2023-10-14T11:58:21.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.986+0000 7f4810f37700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f480c0a8220 con 0x7f480c0c2b10 2023-10-14T11:58:21.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.986+0000 7f4810f37700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f480c1f1770 con 0x7f480c072c80 2023-10-14T11:58:21.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.987+0000 7f480a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f480c0701c0 0x7f480c1f1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.987+0000 7f480a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f480c0701c0 0x7f480c1f1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44676/0 (socket says 172.21.15.78:44676) 2023-10-14T11:58:21.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.987+0000 7f480a59c700 1 -- 172.21.15.78:0/487494857 learned_addr learned my addr 172.21.15.78:0/487494857 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:21.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.988+0000 7f480a59c700 1 -- 172.21.15.78:0/487494857 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f480c072c80 msgr2=0x7f480c1ee600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T11:58:21.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.988+0000 7f480a59c700 1 --2- 172.21.15.78:0/487494857 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f480c072c80 0x7f480c1ee600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.988+0000 7f480a59c700 1 -- 172.21.15.78:0/487494857 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f480c0c2b10 msgr2=0x7f480c0be450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.096 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.988+0000 7f480a59c700 1 --2- 172.21.15.78:0/487494857 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f480c0c2b10 0x7f480c0be450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.096 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.988+0000 7f480a59c700 1 -- 172.21.15.78:0/487494857 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4804009b30 con 0x7f480c0701c0 2023-10-14T11:58:21.096 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.988+0000 7f480a59c700 1 --2- 172.21.15.78:0/487494857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f480c0701c0 0x7f480c1f1230 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f47fc00b500 tx=0x7f47fc00b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.096 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.988+0000 7f47faffd700 1 -- 172.21.15.78:0/487494857 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47fc011830 con 0x7f480c0701c0 2023-10-14T11:58:21.096 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.988+0000 7f4810f37700 1 -- 172.21.15.78:0/487494857 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f480c0bff80 con 0x7f480c0701c0 2023-10-14T11:58:21.097 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.988+0000 7f4810f37700 1 -- 172.21.15.78:0/487494857 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f480c0a7f40 con 0x7f480c0701c0 2023-10-14T11:58:21.097 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.989+0000 7fcf5bfff700 1 --2- 172.21.15.78:0/453434029 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fcf6006af50 0x7fcf6006d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.097 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.989+0000 7fcf5bfff700 1 -- 172.21.15.78:0/453434029 <== mon.1 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcf6808afe0 con 0x7fcf740c0830 2023-10-14T11:58:21.098 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.989+0000 7f2eaf7fe700 1 -- 172.21.15.78:0/1074253911 <== osd.6 v2:172.21.15.195:6808/72554374 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f2e98075b70 con 0x7f2e980730a0 2023-10-14T11:58:21.098 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.989+0000 7fcf5bfff700 1 --2- 172.21.15.78:0/453434029 >> [v2:172.21.15.195:6800/3732756301,v1:172.21.15.195:6801/3732756301] conn(0x7fcf60073170 0x7fcf60075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.098 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.989+0000 7fcf5bfff700 1 -- 172.21.15.78:0/453434029 --> [v2:172.21.15.195:6800/3732756301,v1:172.21.15.195:6801/3732756301] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf60075c40 con 0x7fcf60073170 2023-10-14T11:58:21.099 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.989+0000 7fcf5bfff700 1 -- 172.21.15.78:0/453434029 <== mon.1 v2:172.21.15.195:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf6808b400 con 0x7fcf740c0830 2023-10-14T11:58:21.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.989+0000 7f47faffd700 1 -- 172.21.15.78:0/487494857 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47fc011990 con 0x7f480c0701c0 2023-10-14T11:58:21.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.989+0000 7f47faffd700 1 -- 172.21.15.78:0/487494857 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47fc011d20 con 0x7f480c0701c0 2023-10-14T11:58:21.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.989+0000 7f4810f37700 1 -- 172.21.15.78:0/487494857 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f47f0000fc0 con 0x7f480c0701c0 2023-10-14T11:58:21.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.990+0000 7f8ec58dd700 1 -- 172.21.15.78:0/196405721 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8ec01edf90 con 0x7f8ec00a67d0 2023-10-14T11:58:21.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.990+0000 7fcf737fe700 1 --2- 172.21.15.78:0/453434029 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fcf6006af50 0x7fcf6006d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.991+0000 7fcf737fe700 1 --2- 172.21.15.78:0/453434029 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fcf6006af50 0x7fcf6006d410 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fcf640085a0 tx=0x7fcf640084f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.991+0000 7f47faffd700 1 -- 172.21.15.78:0/487494857 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f47fc01a400 con 0x7f480c0701c0 2023-10-14T11:58:21.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.992+0000 7f47faffd700 1 --2- 172.21.15.78:0/487494857 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f47ec06aed0 0x7f47ec06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.992+0000 7f47faffd700 1 -- 172.21.15.78:0/487494857 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f47fc089dd0 con 0x7f480c0701c0 2023-10-14T11:58:21.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.992+0000 7f47faffd700 1 --2- 172.21.15.78:0/487494857 >> [v2:172.21.15.195:6809/482472979,v1:172.21.15.195:6811/482472979] conn(0x7f47ec073070 0x7f47ec075490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.992+0000 7f47faffd700 1 -- 172.21.15.78:0/487494857 --> [v2:172.21.15.195:6809/482472979,v1:172.21.15.195:6811/482472979] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f47ec075b40 con 0x7f47ec073070 2023-10-14T11:58:21.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.992+0000 7f47faffd700 1 -- 172.21.15.78:0/487494857 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f47fc08a220 con 0x7f480c0701c0 2023-10-14T11:58:21.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.993+0000 7f480ad9d700 1 --2- 172.21.15.78:0/487494857 >> [v2:172.21.15.195:6809/482472979,v1:172.21.15.195:6811/482472979] conn(0x7f47ec073070 0x7f47ec075490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.994+0000 7f480ad9d700 1 --2- 172.21.15.78:0/487494857 >> [v2:172.21.15.195:6809/482472979,v1:172.21.15.195:6811/482472979] conn(0x7f47ec073070 0x7f47ec075490 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.994+0000 7fcf73fff700 1 --2- 172.21.15.78:0/453434029 >> [v2:172.21.15.195:6800/3732756301,v1:172.21.15.195:6801/3732756301] conn(0x7fcf60073170 0x7fcf60075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.995+0000 7fcf73fff700 1 --2- 172.21.15.78:0/453434029 >> [v2:172.21.15.195:6800/3732756301,v1:172.21.15.195:6801/3732756301] conn(0x7fcf60073170 0x7fcf60075590 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.993+0000 7f8ea7fff700 1 -- 172.21.15.78:0/196405721 <== mon.1 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8eb40117d0 con 0x7f8ec00a67d0 2023-10-14T11:58:21.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.993+0000 7f8ea7fff700 1 --2- 172.21.15.78:0/196405721 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8eac06afd0 0x7f8eac06d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.994+0000 7f8ea7fff700 1 -- 172.21.15.78:0/196405721 <== mon.1 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8eb408aa40 con 0x7f8ec00a67d0 2023-10-14T11:58:21.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.994+0000 7f8ea7fff700 1 --2- 172.21.15.78:0/196405721 >> [v2:172.21.15.195:6824/2857706411,v1:172.21.15.195:6825/2857706411] conn(0x7f8eac0731f0 0x7f8eac075610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.994+0000 7f8ea7fff700 1 -- 172.21.15.78:0/196405721 --> [v2:172.21.15.195:6824/2857706411,v1:172.21.15.195:6825/2857706411] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8eac075cc0 con 0x7f8eac0731f0 2023-10-14T11:58:21.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.994+0000 7f8ea7fff700 1 -- 172.21.15.78:0/196405721 <== mon.1 v2:172.21.15.195:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8eb403d070 con 0x7f8ec00a67d0 2023-10-14T11:58:21.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.994+0000 7f8ebeffd700 1 --2- 172.21.15.78:0/196405721 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8eac06afd0 0x7f8eac06d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.995+0000 7f8ebeffd700 1 --2- 172.21.15.78:0/196405721 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8eac06afd0 0x7f8eac06d490 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8eb000bc20 tx=0x7f8eb0013000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.994+0000 7f4809d9b700 1 --2- 172.21.15.78:0/487494857 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f47ec06aed0 0x7f47ec06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.994+0000 7f4809d9b700 1 --2- 172.21.15.78:0/487494857 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f47ec06aed0 0x7f47ec06d390 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f480c0bce50 tx=0x7f4800007a20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.995+0000 7f47faffd700 1 -- 172.21.15.78:0/487494857 <== osd.7 v2:172.21.15.195:6809/482472979 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f47ec075b40 con 0x7f47ec073070 2023-10-14T11:58:21.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.997+0000 7f8ebf7fe700 1 --2- 172.21.15.78:0/196405721 >> [v2:172.21.15.195:6824/2857706411,v1:172.21.15.195:6825/2857706411] conn(0x7f8eac0731f0 0x7f8eac075610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.997+0000 7f8ebf7fe700 1 --2- 172.21.15.78:0/196405721 >> [v2:172.21.15.195:6824/2857706411,v1:172.21.15.195:6825/2857706411] conn(0x7f8eac0731f0 0x7f8eac075610 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:20.998+0000 7f8ea7fff700 1 -- 172.21.15.78:0/196405721 <== osd.5 v2:172.21.15.195:6824/2857706411 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f8eac075cc0 con 0x7f8eac0731f0 2023-10-14T11:58:21.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.001+0000 7fcf5bfff700 1 -- 172.21.15.78:0/453434029 <== osd.4 v2:172.21.15.195:6800/3732756301 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fcf60075c40 con 0x7fcf60073170 2023-10-14T11:58:21.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.010+0000 7f644f08e700 1 -- 172.21.15.78:0/2338394425 --> [v2:172.21.15.78:6800/3884969780,v1:172.21.15.78:6801/3884969780] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6448074b80 con 0x7f642c073070 2023-10-14T11:58:21.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.011+0000 7f64457fa700 1 -- 172.21.15.78:0/2338394425 <== osd.0 v2:172.21.15.78:6800/3884969780 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6448074b80 con 0x7f642c073070 2023-10-14T11:58:21.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.011+0000 7f644f08e700 1 -- 172.21.15.78:0/2338394425 >> [v2:172.21.15.78:6800/3884969780,v1:172.21.15.78:6801/3884969780] conn(0x7f642c073070 msgr2=0x7f642c075490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.011+0000 7f644f08e700 1 --2- 172.21.15.78:0/2338394425 >> [v2:172.21.15.78:6800/3884969780,v1:172.21.15.78:6801/3884969780] conn(0x7f642c073070 0x7f642c075490 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.012+0000 7f644f08e700 1 -- 172.21.15.78:0/2338394425 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f642c06aed0 msgr2=0x7f642c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.012+0000 7f644f08e700 1 --2- 172.21.15.78:0/2338394425 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f642c06aed0 0x7f642c06d390 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f64480719a0 tx=0x7f6438008040 comp rx=0 tx=0).stop 2023-10-14T11:58:21.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.012+0000 7f644f08e700 1 -- 172.21.15.78:0/2338394425 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f64480c0830 msgr2=0x7f64480bdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.012+0000 7f644f08e700 1 --2- 172.21.15.78:0/2338394425 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f64480c0830 0x7f64480bdec0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f643c016000 tx=0x7f643c004750 comp rx=0 tx=0).stop 2023-10-14T11:58:21.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.012+0000 7f644d62b700 1 -- 172.21.15.78:0/2338394425 reap_dead start 2023-10-14T11:58:21.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.012+0000 7f644f08e700 1 -- 172.21.15.78:0/2338394425 shutdown_connections 2023-10-14T11:58:21.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.012+0000 7f644f08e700 1 -- 172.21.15.78:0/2338394425 >> 172.21.15.78:0/2338394425 conn(0x7f644806aab0 msgr2=0x7f6448071c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:21.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.012+0000 7f644f08e700 1 -- 172.21.15.78:0/2338394425 shutdown_connections 2023-10-14T11:58:21.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.012+0000 7f644f08e700 1 -- 172.21.15.78:0/2338394425 wait complete. 2023-10-14T11:58:21.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.013+0000 7fdfca888700 1 -- 172.21.15.78:0/1421665583 --> [v2:172.21.15.78:6808/2176928406,v1:172.21.15.78:6809/2176928406] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdfc40c4920 con 0x7fdfb00731f0 2023-10-14T11:58:21.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.016+0000 7f2ebd0b1700 1 -- 172.21.15.78:0/1074253911 --> [v2:172.21.15.195:6808/72554374,v1:172.21.15.195:6810/72554374] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2e9c002d40 con 0x7f2e980730a0 2023-10-14T11:58:21.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.016+0000 7fdfc0ff9700 1 -- 172.21.15.78:0/1421665583 <== osd.1 v2:172.21.15.78:6808/2176928406 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fdfbc00f4b0 con 0x7fdfb00731f0 2023-10-14T11:58:21.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.017+0000 7fdfca888700 1 -- 172.21.15.78:0/1421665583 >> [v2:172.21.15.78:6808/2176928406,v1:172.21.15.78:6809/2176928406] conn(0x7fdfb00731f0 msgr2=0x7fdfb0075610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.017+0000 7fdfca888700 1 --2- 172.21.15.78:0/1421665583 >> [v2:172.21.15.78:6808/2176928406,v1:172.21.15.78:6809/2176928406] conn(0x7fdfb00731f0 0x7fdfb0075610 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.017+0000 7f2eaf7fe700 1 -- 172.21.15.78:0/1074253911 <== osd.6 v2:172.21.15.195:6808/72554374 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2e9c002d40 con 0x7f2e980730a0 2023-10-14T11:58:21.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.017+0000 7fdfca888700 1 -- 172.21.15.78:0/1421665583 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fdfb006afd0 msgr2=0x7fdfb006d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.017+0000 7fdfca888700 1 --2- 172.21.15.78:0/1421665583 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fdfb006afd0 0x7fdfb006d490 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4009bc0 tx=0x7fdfb4008030 comp rx=0 tx=0).stop 2023-10-14T11:58:21.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.017+0000 7fdfca888700 1 -- 172.21.15.78:0/1421665583 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdfc40a54d0 msgr2=0x7fdfc40b9fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.017+0000 7fdfca888700 1 --2- 172.21.15.78:0/1421665583 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdfc40a54d0 0x7fdfc40b9fe0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdfb8001eb0 tx=0x7fdfb800c6e0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.017+0000 7fdfc8e25700 1 -- 172.21.15.78:0/1421665583 reap_dead start 2023-10-14T11:58:21.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.017+0000 7fdfca888700 1 -- 172.21.15.78:0/1421665583 shutdown_connections 2023-10-14T11:58:21.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.017+0000 7fdfca888700 1 -- 172.21.15.78:0/1421665583 >> 172.21.15.78:0/1421665583 conn(0x7fdfc406aab0 msgr2=0x7fdfc40707d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:21.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.017+0000 7fdfca888700 1 -- 172.21.15.78:0/1421665583 shutdown_connections 2023-10-14T11:58:21.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.017+0000 7fdfca888700 1 -- 172.21.15.78:0/1421665583 wait complete. 2023-10-14T11:58:21.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.019+0000 7f403af66700 1 -- 172.21.15.78:0/1929077980 --> [v2:172.21.15.78:6824/3647044577,v1:172.21.15.78:6825/3647044577] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4014002d40 con 0x7f4020073170 2023-10-14T11:58:21.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.020+0000 7f40317fa700 1 -- 172.21.15.78:0/1929077980 <== osd.2 v2:172.21.15.78:6824/3647044577 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4014002d40 con 0x7f4020073170 2023-10-14T11:58:21.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.020+0000 7f403af66700 1 -- 172.21.15.78:0/1929077980 >> [v2:172.21.15.78:6824/3647044577,v1:172.21.15.78:6825/3647044577] conn(0x7f4020073170 msgr2=0x7f4020075590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.020+0000 7f403af66700 1 --2- 172.21.15.78:0/1929077980 >> [v2:172.21.15.78:6824/3647044577,v1:172.21.15.78:6825/3647044577] conn(0x7f4020073170 0x7f4020075590 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.022+0000 7face62fa700 1 -- 172.21.15.78:0/4166901730 --> [v2:172.21.15.78:6816/3584112288,v1:172.21.15.78:6817/3584112288] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7face01e9ec0 con 0x7faccc073120 2023-10-14T11:58:21.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.024+0000 7f2ebd0b1700 1 -- 172.21.15.78:0/1074253911 >> [v2:172.21.15.195:6808/72554374,v1:172.21.15.195:6810/72554374] conn(0x7f2e980730a0 msgr2=0x7f2e980754c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.024+0000 7f2ebd0b1700 1 --2- 172.21.15.78:0/1074253911 >> [v2:172.21.15.195:6808/72554374,v1:172.21.15.195:6810/72554374] conn(0x7f2e980730a0 0x7f2e980754c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.024+0000 7f2ebd0b1700 1 -- 172.21.15.78:0/1074253911 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f2e9806af00 msgr2=0x7f2e9806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.024+0000 7f2ebd0b1700 1 --2- 172.21.15.78:0/1074253911 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f2e9806af00 0x7f2e9806d3c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8151ab0 tx=0x7f2ea8007ad0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.024+0000 7f2ebd0b1700 1 -- 172.21.15.78:0/1074253911 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2eb8072500 msgr2=0x7f2eb80bf800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.024+0000 7f2ebd0b1700 1 --2- 172.21.15.78:0/1074253911 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2eb8072500 0x7f2eb80bf800 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2ea000b550 tx=0x7f2ea000b910 comp rx=0 tx=0).stop 2023-10-14T11:58:21.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.024+0000 7f2eb759e700 1 -- 172.21.15.78:0/1074253911 reap_dead start 2023-10-14T11:58:21.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.024+0000 7f2ebd0b1700 1 -- 172.21.15.78:0/1074253911 shutdown_connections 2023-10-14T11:58:21.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.024+0000 7f2ebd0b1700 1 -- 172.21.15.78:0/1074253911 >> 172.21.15.78:0/1074253911 conn(0x7f2eb806aab0 msgr2=0x7f2eb80a7820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:21.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.024+0000 7facdcff9700 1 -- 172.21.15.78:0/4166901730 <== osd.3 v2:172.21.15.78:6816/3584112288 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7facd8003a20 con 0x7faccc073120 2023-10-14T11:58:21.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.024+0000 7f4810f37700 1 -- 172.21.15.78:0/487494857 --> [v2:172.21.15.195:6809/482472979,v1:172.21.15.195:6811/482472979] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f47f0002d40 con 0x7f47ec073070 2023-10-14T11:58:21.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.024+0000 7f403af66700 1 -- 172.21.15.78:0/1929077980 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f402006af50 msgr2=0x7f402006d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.024+0000 7f403af66700 1 --2- 172.21.15.78:0/1929077980 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f402006af50 0x7f402006d410 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f402c009fd0 tx=0x7f402c00b040 comp rx=0 tx=0).stop 2023-10-14T11:58:21.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.024+0000 7f403af66700 1 -- 172.21.15.78:0/1929077980 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f40340c59e0 msgr2=0x7f40340a2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.024+0000 7f403af66700 1 --2- 172.21.15.78:0/1929077980 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f40340c59e0 0x7f40340a2240 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4028000f60 tx=0x7f402800c670 comp rx=0 tx=0).stop 2023-10-14T11:58:21.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.024+0000 7f4039503700 1 -- 172.21.15.78:0/1929077980 reap_dead start 2023-10-14T11:58:21.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.025+0000 7face62fa700 1 -- 172.21.15.78:0/4166901730 >> [v2:172.21.15.78:6816/3584112288,v1:172.21.15.78:6817/3584112288] conn(0x7faccc073120 msgr2=0x7faccc075540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.025+0000 7face62fa700 1 --2- 172.21.15.78:0/4166901730 >> [v2:172.21.15.78:6816/3584112288,v1:172.21.15.78:6817/3584112288] conn(0x7faccc073120 0x7faccc075540 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.025+0000 7f47faffd700 1 -- 172.21.15.78:0/487494857 <== osd.7 v2:172.21.15.195:6809/482472979 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f47f0002d40 con 0x7f47ec073070 2023-10-14T11:58:21.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.025+0000 7f4810f37700 1 -- 172.21.15.78:0/487494857 >> [v2:172.21.15.195:6809/482472979,v1:172.21.15.195:6811/482472979] conn(0x7f47ec073070 msgr2=0x7f47ec075490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.025+0000 7f4810f37700 1 --2- 172.21.15.78:0/487494857 >> [v2:172.21.15.195:6809/482472979,v1:172.21.15.195:6811/482472979] conn(0x7f47ec073070 0x7f47ec075490 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.026+0000 7f4810f37700 1 -- 172.21.15.78:0/487494857 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f47ec06aed0 msgr2=0x7f47ec06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.026+0000 7f4810f37700 1 --2- 172.21.15.78:0/487494857 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f47ec06aed0 0x7f47ec06d390 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f480c0bce50 tx=0x7f4800007a20 comp rx=0 tx=0).stop 2023-10-14T11:58:21.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.026+0000 7f4810f37700 1 -- 172.21.15.78:0/487494857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f480c0701c0 msgr2=0x7f480c1f1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.026+0000 7f4810f37700 1 --2- 172.21.15.78:0/487494857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f480c0701c0 0x7f480c1f1230 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f47fc00b500 tx=0x7f47fc00b8c0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.026+0000 7f403af66700 1 -- 172.21.15.78:0/1929077980 shutdown_connections 2023-10-14T11:58:21.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.026+0000 7f403af66700 1 -- 172.21.15.78:0/1929077980 >> 172.21.15.78:0/1929077980 conn(0x7f403406aab0 msgr2=0x7f4034071800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:21.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.026+0000 7f2ebd0b1700 1 -- 172.21.15.78:0/1074253911 shutdown_connections 2023-10-14T11:58:21.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.026+0000 7f480ad9d700 1 -- 172.21.15.78:0/487494857 reap_dead start 2023-10-14T11:58:21.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.026+0000 7f2ebd0b1700 1 -- 172.21.15.78:0/1074253911 wait complete. 2023-10-14T11:58:21.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.026+0000 7f4810f37700 1 -- 172.21.15.78:0/487494857 shutdown_connections 2023-10-14T11:58:21.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.026+0000 7f4810f37700 1 -- 172.21.15.78:0/487494857 >> 172.21.15.78:0/487494857 conn(0x7f480c06aab0 msgr2=0x7f480c1e50c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:21.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.026+0000 7f4810f37700 1 -- 172.21.15.78:0/487494857 shutdown_connections 2023-10-14T11:58:21.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.026+0000 7f4810f37700 1 -- 172.21.15.78:0/487494857 wait complete. 2023-10-14T11:58:21.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.026+0000 7f403af66700 1 -- 172.21.15.78:0/1929077980 shutdown_connections 2023-10-14T11:58:21.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.026+0000 7f403af66700 1 -- 172.21.15.78:0/1929077980 wait complete. 2023-10-14T11:58:21.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.027+0000 7face62fa700 1 -- 172.21.15.78:0/4166901730 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7faccc06af00 msgr2=0x7faccc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.027+0000 7face62fa700 1 --2- 172.21.15.78:0/4166901730 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7faccc06af00 0x7faccc06d3c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7face0072540 tx=0x7facd000a320 comp rx=0 tx=0).stop 2023-10-14T11:58:21.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.027+0000 7face62fa700 1 -- 172.21.15.78:0/4166901730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7face00706b0 msgr2=0x7face00a6a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.027+0000 7face62fa700 1 --2- 172.21.15.78:0/4166901730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7face00706b0 0x7face00a6a70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7facd400b620 tx=0x7facd400b930 comp rx=0 tx=0).stop 2023-10-14T11:58:21.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.028+0000 7face4897700 1 -- 172.21.15.78:0/4166901730 reap_dead start 2023-10-14T11:58:21.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.027+0000 7f8ec58dd700 1 -- 172.21.15.78:0/196405721 --> [v2:172.21.15.195:6824/2857706411,v1:172.21.15.195:6825/2857706411] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8ec00a7f80 con 0x7f8eac0731f0 2023-10-14T11:58:21.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.028+0000 7face62fa700 1 -- 172.21.15.78:0/4166901730 shutdown_connections 2023-10-14T11:58:21.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.028+0000 7face62fa700 1 -- 172.21.15.78:0/4166901730 >> 172.21.15.78:0/4166901730 conn(0x7face006aab0 msgr2=0x7face0152960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:21.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.028+0000 7face62fa700 1 -- 172.21.15.78:0/4166901730 shutdown_connections 2023-10-14T11:58:21.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.028+0000 7face62fa700 1 -- 172.21.15.78:0/4166901730 wait complete. 2023-10-14T11:58:21.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.030+0000 7f8ea7fff700 1 -- 172.21.15.78:0/196405721 <== osd.5 v2:172.21.15.195:6824/2857706411 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8eb8013390 con 0x7f8eac0731f0 2023-10-14T11:58:21.136 INFO:teuthology.orchestra.run.smithi078.stdout:55834574852 2023-10-14T11:58:21.137 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-14T11:58:21.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.034+0000 7fcf7a1c2700 1 -- 172.21.15.78:0/453434029 --> [v2:172.21.15.195:6800/3732756301,v1:172.21.15.195:6801/3732756301] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcf74074b80 con 0x7fcf60073170 2023-10-14T11:58:21.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.035+0000 7f8ec58dd700 1 -- 172.21.15.78:0/196405721 >> [v2:172.21.15.195:6824/2857706411,v1:172.21.15.195:6825/2857706411] conn(0x7f8eac0731f0 msgr2=0x7f8eac075610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.035+0000 7f8ec58dd700 1 --2- 172.21.15.78:0/196405721 >> [v2:172.21.15.195:6824/2857706411,v1:172.21.15.195:6825/2857706411] conn(0x7f8eac0731f0 0x7f8eac075610 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.035+0000 7fcf5bfff700 1 -- 172.21.15.78:0/453434029 <== osd.4 v2:172.21.15.195:6800/3732756301 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fcf74074b80 con 0x7fcf60073170 2023-10-14T11:58:21.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.035+0000 7f8ec58dd700 1 -- 172.21.15.78:0/196405721 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8eac06afd0 msgr2=0x7f8eac06d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.035+0000 7f8ec58dd700 1 --2- 172.21.15.78:0/196405721 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8eac06afd0 0x7f8eac06d490 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8eb000bc20 tx=0x7f8eb0013000 comp rx=0 tx=0).stop 2023-10-14T11:58:21.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.035+0000 7f8ec58dd700 1 -- 172.21.15.78:0/196405721 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8ec00a67d0 msgr2=0x7f8ec01ecfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.035+0000 7f8ec58dd700 1 --2- 172.21.15.78:0/196405721 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8ec00a67d0 0x7f8ec01ecfc0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4010d60 tx=0x7f8eb400e620 comp rx=0 tx=0).stop 2023-10-14T11:58:21.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.035+0000 7f8ebf7fe700 1 -- 172.21.15.78:0/196405721 reap_dead start 2023-10-14T11:58:21.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.036+0000 7fcf7a1c2700 1 -- 172.21.15.78:0/453434029 >> [v2:172.21.15.195:6800/3732756301,v1:172.21.15.195:6801/3732756301] conn(0x7fcf60073170 msgr2=0x7fcf60075590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.036+0000 7fcf7a1c2700 1 --2- 172.21.15.78:0/453434029 >> [v2:172.21.15.195:6800/3732756301,v1:172.21.15.195:6801/3732756301] conn(0x7fcf60073170 0x7fcf60075590 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.039+0000 7f8ec58dd700 1 -- 172.21.15.78:0/196405721 shutdown_connections 2023-10-14T11:58:21.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.039+0000 7f8ec58dd700 1 -- 172.21.15.78:0/196405721 >> 172.21.15.78:0/196405721 conn(0x7f8ec006aab0 msgr2=0x7f8ec0152f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:21.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.039+0000 7fcf7a1c2700 1 -- 172.21.15.78:0/453434029 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fcf6006af50 msgr2=0x7fcf6006d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.039+0000 7fcf7a1c2700 1 --2- 172.21.15.78:0/453434029 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fcf6006af50 0x7fcf6006d410 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fcf640085a0 tx=0x7fcf640084f0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.039+0000 7fcf7a1c2700 1 -- 172.21.15.78:0/453434029 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcf740c0830 msgr2=0x7fcf7414d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.039+0000 7fcf7a1c2700 1 --2- 172.21.15.78:0/453434029 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcf740c0830 0x7fcf7414d7e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcf68009b60 tx=0x7fcf68004750 comp rx=0 tx=0).stop 2023-10-14T11:58:21.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.039+0000 7fcf73fff700 1 -- 172.21.15.78:0/453434029 reap_dead start 2023-10-14T11:58:21.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.039+0000 7fcf7a1c2700 1 -- 172.21.15.78:0/453434029 shutdown_connections 2023-10-14T11:58:21.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.039+0000 7fcf7a1c2700 1 -- 172.21.15.78:0/453434029 >> 172.21.15.78:0/453434029 conn(0x7fcf7406aab0 msgr2=0x7fcf7406ae90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:21.144 INFO:teuthology.orchestra.run.smithi078.stdout:55834574853 2023-10-14T11:58:21.144 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-14T11:58:21.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.039+0000 7fcf7a1c2700 1 -- 172.21.15.78:0/453434029 shutdown_connections 2023-10-14T11:58:21.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.039+0000 7fcf7a1c2700 1 -- 172.21.15.78:0/453434029 wait complete. 2023-10-14T11:58:21.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.042+0000 7f8ec58dd700 1 -- 172.21.15.78:0/196405721 shutdown_connections 2023-10-14T11:58:21.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.042+0000 7f8ec58dd700 1 -- 172.21.15.78:0/196405721 wait complete. 2023-10-14T11:58:21.146 INFO:teuthology.orchestra.run.smithi078.stdout:55834574853 2023-10-14T11:58:21.146 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-14T11:58:21.147 INFO:teuthology.orchestra.run.smithi078.stdout:55834574852 2023-10-14T11:58:21.148 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-14T11:58:21.149 INFO:teuthology.orchestra.run.smithi078.stdout:55834574853 2023-10-14T11:58:21.149 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-14T11:58:21.150 INFO:teuthology.orchestra.run.smithi078.stdout:55834574853 2023-10-14T11:58:21.150 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-14T11:58:21.151 INFO:teuthology.orchestra.run.smithi078.stdout:55834574853 2023-10-14T11:58:21.152 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-14T11:58:21.153 INFO:teuthology.orchestra.run.smithi078.stdout:55834574853 2023-10-14T11:58:21.153 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-14T11:58:21.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.388+0000 7fd685f22700 1 Processor -- start 2023-10-14T11:58:21.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.388+0000 7fd685f22700 1 -- start start 2023-10-14T11:58:21.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.389+0000 7fd685f22700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd6800c2ed0 0x7fd6800c3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.389+0000 7fd685f22700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd68006df80 0x7fd6800c4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.390+0000 7fd685f22700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6800b9760 con 0x7fd6800c4a50 2023-10-14T11:58:21.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.390+0000 7fd685f22700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd68008a9a0 con 0x7fd6800c2ed0 2023-10-14T11:58:21.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.390+0000 7fd685f22700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6800c0830 con 0x7fd68006df80 2023-10-14T11:58:21.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.390+0000 7fd67ffff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fd6800c4a50 0x7fd6800a7930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.78:39376/0 (socket says 172.21.15.78:39376) 2023-10-14T11:58:21.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.390+0000 7fd67ffff700 1 -- 172.21.15.78:0/265416773 learned_addr learned my addr 172.21.15.78:0/265416773 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:21.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.390+0000 7fd67f7fe700 1 --2- 172.21.15.78:0/265416773 >> v2:172.21.15.78:3300/0 conn(0x7fd6800c2ed0 0x7fd6800c3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.391+0000 7fd67f7fe700 1 -- 172.21.15.78:0/265416773 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd68006df80 msgr2=0x7fd6800c4510 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-14T11:58:21.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.391+0000 7fd67f7fe700 1 --2- 172.21.15.78:0/265416773 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd68006df80 0x7fd6800c4510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.391+0000 7fd67f7fe700 1 -- 172.21.15.78:0/265416773 >> v1:172.21.15.195:6789/0 conn(0x7fd6800c4a50 legacy=0x7fd6800a7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.391+0000 7fd67f7fe700 1 -- 172.21.15.78:0/265416773 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd68008abf0 con 0x7fd6800c2ed0 2023-10-14T11:58:21.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.391+0000 7fd67f7fe700 1 --2- 172.21.15.78:0/265416773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6800c2ed0 0x7fd6800c3fd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd67800d1e0 tx=0x7fd67800d4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b154cdde27dedf42 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.391+0000 7fd67e7fc700 1 -- 172.21.15.78:0/265416773 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd678010070 con 0x7fd6800c2ed0 2023-10-14T11:58:21.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.391+0000 7fd67e7fc700 1 -- 172.21.15.78:0/265416773 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd67800dba0 con 0x7fd6800c2ed0 2023-10-14T11:58:21.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.391+0000 7fd67e7fc700 1 -- 172.21.15.78:0/265416773 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd678007aa0 con 0x7fd6800c2ed0 2023-10-14T11:58:21.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.391+0000 7f45c1aa4700 1 Processor -- start 2023-10-14T11:58:21.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.392+0000 7fd685f22700 1 -- 172.21.15.78:0/265416773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6800c2ed0 msgr2=0x7fd6800c3fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.392+0000 7fd685f22700 1 --2- 172.21.15.78:0/265416773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6800c2ed0 0x7fd6800c3fd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd67800d1e0 tx=0x7fd67800d4f0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.392+0000 7f45c1aa4700 1 -- start start 2023-10-14T11:58:21.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.393+0000 7f45c1aa4700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f45bc0a7ba0 0x7f45bc072a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.393+0000 7f45c1aa4700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f45bc0c0830 0x7f45bc0c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.393+0000 7f45c1aa4700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f45bc0c59e0 0x7f45bc06dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.393+0000 7f45c1aa4700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45bc0c18b0 con 0x7f45bc0c0830 2023-10-14T11:58:21.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.393+0000 7f45c1aa4700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45bc0c15d0 con 0x7f45bc0a7ba0 2023-10-14T11:58:21.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.393+0000 7f45c1aa4700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45bc0c3f30 con 0x7f45bc0c59e0 2023-10-14T11:58:21.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.393+0000 7fd685f22700 1 -- 172.21.15.78:0/265416773 shutdown_connections 2023-10-14T11:58:21.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.393+0000 7fd685f22700 1 --2- 172.21.15.78:0/265416773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6800c2ed0 0x7fd6800c3fd0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.393+0000 7fd685f22700 1 --2- 172.21.15.78:0/265416773 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd68006df80 0x7fd6800c4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.393+0000 7fd685f22700 1 -- 172.21.15.78:0/265416773 >> 172.21.15.78:0/265416773 conn(0x7fd68006aab0 msgr2=0x7fd68006aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:21.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.388+0000 7f4140530700 1 Processor -- start 2023-10-14T11:58:21.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.388+0000 7f4140530700 1 -- start start 2023-10-14T11:58:21.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.390+0000 7f4140530700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f41380c2ed0 0x7f41380c3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.390+0000 7f4140530700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f41380a4ea0 0x7f41381657c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.390+0000 7f4140530700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41380b9760 con 0x7f413806df80 2023-10-14T11:58:21.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.390+0000 7f4140530700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f413808a9a0 con 0x7f41380c2ed0 2023-10-14T11:58:21.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.393+0000 7f45baffd700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f45bc0c0830 0x7f45bc0c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.393+0000 7f45baffd700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f45bc0c0830 0x7f45bc0c54a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44708/0 (socket says 172.21.15.78:44708) 2023-10-14T11:58:21.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.393+0000 7f45baffd700 1 -- 172.21.15.78:0/2552120787 learned_addr learned my addr 172.21.15.78:0/2552120787 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:21.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.390+0000 7f4140530700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41380c0830 con 0x7f41380a4ea0 2023-10-14T11:58:21.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.390+0000 7f413eacd700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f41380a4ea0 0x7f41381657c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.390+0000 7f413eacd700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f41380a4ea0 0x7f41381657c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.78:44364/0 (socket says 172.21.15.78:44364) 2023-10-14T11:58:21.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.390+0000 7f413eacd700 1 -- 172.21.15.78:0/2181582495 learned_addr learned my addr 172.21.15.78:0/2181582495 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:21.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.391+0000 7f413d2ca700 1 -- 172.21.15.78:0/2181582495 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3731767599 0 0) 0x7f41380b9760 con 0x7f413806df80 2023-10-14T11:58:21.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.393+0000 7f413d2ca700 1 -- 172.21.15.78:0/2181582495 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4124003590 con 0x7f413806df80 2023-10-14T11:58:21.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.393+0000 7f45baffd700 1 -- 172.21.15.78:0/2552120787 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f45bc0c59e0 msgr2=0x7f45bc06dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:21.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.393+0000 7f45baffd700 1 --2- 172.21.15.78:0/2552120787 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f45bc0c59e0 0x7f45bc06dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.393+0000 7f45baffd700 1 -- 172.21.15.78:0/2552120787 >> v2:172.21.15.195:3300/0 conn(0x7f45bc0a7ba0 msgr2=0x7f45bc072a70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-14T11:58:21.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.393+0000 7f45baffd700 1 --2- 172.21.15.78:0/2552120787 >> v2:172.21.15.195:3300/0 conn(0x7f45bc0a7ba0 0x7f45bc072a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.393+0000 7f45baffd700 1 -- 172.21.15.78:0/2552120787 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45bc0c4210 con 0x7f45bc0c0830 2023-10-14T11:58:21.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.394+0000 7f45baffd700 1 --2- 172.21.15.78:0/2552120787 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45bc0c0830 0x7f45bc0c54a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f45bc0a8a20 tx=0x7f45b0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b26a8219135f2a6d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.394+0000 7fd685f22700 1 -- 172.21.15.78:0/265416773 shutdown_connections 2023-10-14T11:58:21.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.394+0000 7f45ba7fc700 1 -- 172.21.15.78:0/2552120787 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b000b870 con 0x7f45bc0c0830 2023-10-14T11:58:21.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.394+0000 7f45ba7fc700 1 -- 172.21.15.78:0/2552120787 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45b000b9d0 con 0x7f45bc0c0830 2023-10-14T11:58:21.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.394+0000 7f45ba7fc700 1 -- 172.21.15.78:0/2552120787 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b000bd60 con 0x7f45bc0c0830 2023-10-14T11:58:21.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.395+0000 7f45c1aa4700 1 -- 172.21.15.78:0/2552120787 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45bc0c0830 msgr2=0x7f45bc0c54a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.395+0000 7f45c1aa4700 1 --2- 172.21.15.78:0/2552120787 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45bc0c0830 0x7f45bc0c54a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f45bc0a8a20 tx=0x7f45b0009bd0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.395+0000 7f45c1aa4700 1 -- 172.21.15.78:0/2552120787 shutdown_connections 2023-10-14T11:58:21.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.395+0000 7f45c1aa4700 1 --2- 172.21.15.78:0/2552120787 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f45bc0c59e0 0x7f45bc06dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.395+0000 7f45c1aa4700 1 --2- 172.21.15.78:0/2552120787 >> v2:172.21.15.195:3300/0 conn(0x7f45bc0a7ba0 0x7f45bc072a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.395+0000 7f45c1aa4700 1 --2- 172.21.15.78:0/2552120787 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45bc0c0830 0x7f45bc0c54a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.395+0000 7f45c1aa4700 1 -- 172.21.15.78:0/2552120787 >> 172.21.15.78:0/2552120787 conn(0x7f45bc06aab0 msgr2=0x7f45bc06aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:21.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.394+0000 7fd685f22700 1 -- 172.21.15.78:0/265416773 wait complete. 2023-10-14T11:58:21.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.393+0000 7f413eacd700 1 -- 172.21.15.78:0/2181582495 >> v2:172.21.15.78:3300/0 conn(0x7f41380c2ed0 msgr2=0x7f41380c3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:21.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.393+0000 7f413d2ca700 1 -- 172.21.15.78:0/2181582495 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2735782900 0 0) 0x7f4124003590 con 0x7f413806df80 2023-10-14T11:58:21.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.390+0000 7f413e2cc700 1 --2- 172.21.15.78:0/2181582495 >> v2:172.21.15.78:3300/0 conn(0x7f41380c2ed0 0x7f41380c3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.394+0000 7f413eacd700 1 --2- 172.21.15.78:0/2181582495 >> v2:172.21.15.78:3300/0 conn(0x7f41380c2ed0 0x7f41380c3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.394+0000 7f413eacd700 1 -- 172.21.15.78:0/2181582495 >> v1:172.21.15.78:6789/0 conn(0x7f413806df80 legacy=0x7f41380c4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.394+0000 7f413eacd700 1 -- 172.21.15.78:0/2181582495 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f413808abf0 con 0x7f41380a4ea0 2023-10-14T11:58:21.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.395+0000 7f413e2cc700 1 --2- 172.21.15.78:0/2181582495 >> v2:172.21.15.78:3300/0 conn(0x7f41380c2ed0 0x7f41380c3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:21.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.395+0000 7f45c1aa4700 1 -- 172.21.15.78:0/2552120787 shutdown_connections 2023-10-14T11:58:21.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.395+0000 7fd685f22700 1 Processor -- start 2023-10-14T11:58:21.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.395+0000 7f45c1aa4700 1 -- 172.21.15.78:0/2552120787 wait complete. 2023-10-14T11:58:21.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.396+0000 7f413eacd700 1 --2- 172.21.15.78:0/2181582495 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f41380a4ea0 0x7f41381657c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4130009e50 tx=0x7f413000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2682cb0e0fb786d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.396+0000 7fd685f22700 1 -- start start 2023-10-14T11:58:21.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.397+0000 7fd685f22700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6800c2ed0 0x7fd6800b9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.397+0000 7fd685f22700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd6800c4a50 0x7fd6800ba960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.397+0000 7fd685f22700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd680089e70 0x7fd6800be760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.397+0000 7fd685f22700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6800ba540 con 0x7fd6800c2ed0 2023-10-14T11:58:21.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.397+0000 7fd685f22700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6800baea0 con 0x7fd6800c4a50 2023-10-14T11:58:21.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.397+0000 7fd685f22700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6801f95a0 con 0x7fd680089e70 2023-10-14T11:58:21.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.397+0000 7fd67f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6800c2ed0 0x7fd6800b9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.397+0000 7fd67f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6800c2ed0 0x7fd6800b9930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44710/0 (socket says 172.21.15.78:44710) 2023-10-14T11:58:21.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.397+0000 7fd67f7fe700 1 -- 172.21.15.78:0/3126224691 learned_addr learned my addr 172.21.15.78:0/3126224691 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:21.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.397+0000 7f413d2ca700 1 -- 172.21.15.78:0/2181582495 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f413000e070 con 0x7f41380a4ea0 2023-10-14T11:58:21.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.397+0000 7f413d2ca700 1 -- 172.21.15.78:0/2181582495 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4130003a20 con 0x7f41380a4ea0 2023-10-14T11:58:21.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.397+0000 7f413d2ca700 1 -- 172.21.15.78:0/2181582495 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4128001670 con 0x7f41380a4ea0 2023-10-14T11:58:21.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.397+0000 7fd67f7fe700 1 -- 172.21.15.78:0/3126224691 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd680089e70 msgr2=0x7fd6800be760 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-14T11:58:21.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.397+0000 7fd67f7fe700 1 --2- 172.21.15.78:0/3126224691 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd680089e70 0x7fd6800be760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.397+0000 7fd67f7fe700 1 -- 172.21.15.78:0/3126224691 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd6800c4a50 msgr2=0x7fd6800ba960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.397+0000 7fd67f7fe700 1 --2- 172.21.15.78:0/3126224691 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd6800c4a50 0x7fd6800ba960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.397+0000 7fd67f7fe700 1 -- 172.21.15.78:0/3126224691 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd678007910 con 0x7fd6800c2ed0 2023-10-14T11:58:21.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.398+0000 7fd67f7fe700 1 --2- 172.21.15.78:0/3126224691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6800c2ed0 0x7fd6800b9930 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd67800d8f0 tx=0x7fd678004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.398+0000 7fd667fff700 1 -- 172.21.15.78:0/3126224691 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd678004740 con 0x7fd6800c2ed0 2023-10-14T11:58:21.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.398+0000 7fd667fff700 1 -- 172.21.15.78:0/3126224691 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6780048c0 con 0x7fd6800c2ed0 2023-10-14T11:58:21.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.398+0000 7fd667fff700 1 -- 172.21.15.78:0/3126224691 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd678008270 con 0x7fd6800c2ed0 2023-10-14T11:58:21.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.397+0000 7f45c1aa4700 1 Processor -- start 2023-10-14T11:58:21.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.398+0000 7f45c1aa4700 1 -- start start 2023-10-14T11:58:21.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.398+0000 7f4140530700 1 -- 172.21.15.78:0/2181582495 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f41380a4ea0 msgr2=0x7f41381657c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.398+0000 7f4140530700 1 --2- 172.21.15.78:0/2181582495 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f41380a4ea0 0x7f41381657c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4130009e50 tx=0x7f413000b330 comp rx=0 tx=0).stop 2023-10-14T11:58:21.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.398+0000 7f4140530700 1 -- 172.21.15.78:0/2181582495 shutdown_connections 2023-10-14T11:58:21.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.398+0000 7f4140530700 1 --2- 172.21.15.78:0/2181582495 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f41380a4ea0 0x7f41381657c0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.398+0000 7f4140530700 1 --2- 172.21.15.78:0/2181582495 >> v2:172.21.15.78:3300/0 conn(0x7f41380c2ed0 0x7f41380c3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.398+0000 7f4140530700 1 -- 172.21.15.78:0/2181582495 >> 172.21.15.78:0/2181582495 conn(0x7f413806aab0 msgr2=0x7f413806aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:21.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.398+0000 7fd685f22700 1 -- 172.21.15.78:0/3126224691 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6800b9f10 con 0x7fd6800c2ed0 2023-10-14T11:58:21.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.398+0000 7f45c1aa4700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f45bc0c0830 0x7f45bc1ed3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.398+0000 7f45c1aa4700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f45bc0c59e0 0x7f45bc0aa650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.398+0000 7f45c1aa4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45bc0a9a50 0x7f45bc0aab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.398+0000 7fd685f22700 1 -- 172.21.15.78:0/3126224691 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6800bf890 con 0x7fd6800c2ed0 2023-10-14T11:58:21.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.398+0000 7f45c1aa4700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45bc0a9000 con 0x7f45bc0a9a50 2023-10-14T11:58:21.424 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.398+0000 7f45c1aa4700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45bc0aa200 con 0x7f45bc0c59e0 2023-10-14T11:58:21.424 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.398+0000 7f45c1aa4700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45bc0b9e60 con 0x7f45bc0c0830 2023-10-14T11:58:21.424 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.399+0000 7fd52e8e6700 1 Processor -- start 2023-10-14T11:58:21.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.399+0000 7f5a1bace700 1 Processor -- start 2023-10-14T11:58:21.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.399+0000 7f45bb7fe700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f45bc0c0830 0x7f45bc1ed3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.399+0000 7f45bb7fe700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f45bc0c0830 0x7f45bc1ed3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.78:44386/0 (socket says 172.21.15.78:44386) 2023-10-14T11:58:21.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.399+0000 7f45bb7fe700 1 -- 172.21.15.78:0/2593509420 learned_addr learned my addr 172.21.15.78:0/2593509420 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:21.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.399+0000 7f45bb7fe700 1 -- 172.21.15.78:0/2593509420 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f45bc0c59e0 msgr2=0x7f45bc0aa650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.399+0000 7f45bb7fe700 1 --2- 172.21.15.78:0/2593509420 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f45bc0c59e0 0x7f45bc0aa650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.399+0000 7f45bb7fe700 1 -- 172.21.15.78:0/2593509420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45bc0a9a50 msgr2=0x7f45bc0aab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.427 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.399+0000 7f45bb7fe700 1 --2- 172.21.15.78:0/2593509420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45bc0a9a50 0x7f45bc0aab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.427 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.399+0000 7f45bb7fe700 1 -- 172.21.15.78:0/2593509420 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b0009550 con 0x7f45bc0c0830 2023-10-14T11:58:21.427 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.399+0000 7f45bb7fe700 1 --2- 172.21.15.78:0/2593509420 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f45bc0c0830 0x7f45bc1ed3f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f45ac00b4a0 tx=0x7f45ac00b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.427 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.399+0000 7f45a3fff700 1 -- 172.21.15.78:0/2593509420 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ac014e30 con 0x7f45bc0c0830 2023-10-14T11:58:21.427 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.399+0000 7f45a3fff700 1 -- 172.21.15.78:0/2593509420 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45ac01c440 con 0x7f45bc0c0830 2023-10-14T11:58:21.428 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.399+0000 7f45a3fff700 1 -- 172.21.15.78:0/2593509420 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ac01c7f0 con 0x7f45bc0c0830 2023-10-14T11:58:21.428 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.400+0000 7f5a1bace700 1 -- start start 2023-10-14T11:58:21.429 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.401+0000 7f5a1bace700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5a140a7ba0 0x7f5a14072a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.429 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.401+0000 7f5a1bace700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f5a140c0830 0x7f5a140c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.429 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.401+0000 7f5a1bace700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a140c18b0 con 0x7f5a140c59e0 2023-10-14T11:58:21.429 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.401+0000 7f5a1bace700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a140c15d0 con 0x7f5a140a7ba0 2023-10-14T11:58:21.429 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.401+0000 7f5a1bace700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a140c3f30 con 0x7f5a140c0830 2023-10-14T11:58:21.430 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.401+0000 7f5a1a06b700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f5a140c59e0 0x7f5a1406d220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49598/0 (socket says 172.21.15.78:49598) 2023-10-14T11:58:21.430 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.401+0000 7f5a1a06b700 1 -- 172.21.15.78:0/3948226941 learned_addr learned my addr 172.21.15.78:0/3948226941 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:21.430 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.401+0000 7f5a1986a700 1 --2- 172.21.15.78:0/3948226941 >> v2:172.21.15.78:3300/0 conn(0x7f5a140a7ba0 0x7f5a14072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.431 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.400+0000 7fd52e8e6700 1 -- start start 2023-10-14T11:58:21.431 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.401+0000 7fd52e8e6700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd5280c0830 0x7fd528072a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.431 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.401+0000 7fd52e8e6700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5280706b0 0x7fd5281548c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.431 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.401+0000 7fd52e8e6700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5280c18b0 con 0x7fd5280a7ba0 2023-10-14T11:58:21.431 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.401+0000 7fd52e8e6700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5280c15d0 con 0x7fd5280c0830 2023-10-14T11:58:21.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.401+0000 7fd52e8e6700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5280c3f30 con 0x7fd5280706b0 2023-10-14T11:58:21.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.402+0000 7fd527fff700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd5280a7ba0 0x7fd5280c54a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49604/0 (socket says 172.21.15.78:49604) 2023-10-14T11:58:21.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.400+0000 7f45c1aa4700 1 -- 172.21.15.78:0/2593509420 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45bc1edd30 con 0x7f45bc0c0830 2023-10-14T11:58:21.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.400+0000 7f45c1aa4700 1 -- 172.21.15.78:0/2593509420 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45bc0ba3f0 con 0x7f45bc0c0830 2023-10-14T11:58:21.433 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.402+0000 7fd527fff700 1 -- 172.21.15.78:0/4275551362 learned_addr learned my addr 172.21.15.78:0/4275551362 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:21.433 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.402+0000 7fd52ce83700 1 --2- 172.21.15.78:0/4275551362 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5280706b0 0x7fd5281548c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.402+0000 7f5a19069700 1 --2- 172.21.15.78:0/3948226941 >> v2:172.21.15.195:3300/0 conn(0x7f5a140c0830 0x7f5a140c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.402+0000 7f5a18868700 1 -- 172.21.15.78:0/3948226941 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1962852459 0 0) 0x7f5a140c18b0 con 0x7f5a140c59e0 2023-10-14T11:58:21.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.402+0000 7f5a18868700 1 -- 172.21.15.78:0/3948226941 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a000057c0 con 0x7f5a140c59e0 2023-10-14T11:58:21.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.402+0000 7f5a1986a700 1 -- 172.21.15.78:0/3948226941 >> v2:172.21.15.195:3300/0 conn(0x7f5a140c0830 msgr2=0x7f5a140c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:21.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.402+0000 7f5a1986a700 1 --2- 172.21.15.78:0/3948226941 >> v2:172.21.15.195:3300/0 conn(0x7f5a140c0830 0x7f5a140c54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.402+0000 7f5a1986a700 1 -- 172.21.15.78:0/3948226941 >> v1:172.21.15.78:6789/0 conn(0x7f5a140c59e0 legacy=0x7f5a1406d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.402+0000 7fd52ce83700 1 -- 172.21.15.78:0/4275551362 >> v2:172.21.15.78:3300/0 conn(0x7fd5280c0830 msgr2=0x7fd528072a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:21.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.402+0000 7fd52ce83700 1 --2- 172.21.15.78:0/4275551362 >> v2:172.21.15.78:3300/0 conn(0x7fd5280c0830 0x7fd528072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.402+0000 7fd52ce83700 1 -- 172.21.15.78:0/4275551362 >> v1:172.21.15.78:6789/0 conn(0x7fd5280a7ba0 legacy=0x7fd5280c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.402+0000 7fd52ce83700 1 -- 172.21.15.78:0/4275551362 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5280c4210 con 0x7fd5280706b0 2023-10-14T11:58:21.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.402+0000 7f5a1986a700 1 -- 172.21.15.78:0/3948226941 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a140c4210 con 0x7f5a140a7ba0 2023-10-14T11:58:21.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.402+0000 7fd52ce83700 1 --2- 172.21.15.78:0/4275551362 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5280706b0 0x7fd5281548c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd52000c6a0 tx=0x7fd52000c9b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78d6d97e5f685e43 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.402+0000 7f45a3fff700 1 -- 172.21.15.78:0/2593509420 <== mon.2 v2:172.21.15.195:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f45ac01c950 con 0x7f45bc0c0830 2023-10-14T11:58:21.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.402+0000 7fd685f22700 1 -- 172.21.15.78:0/3126224691 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6800ba680 con 0x7fd6800c2ed0 2023-10-14T11:58:21.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.402+0000 7f45c1aa4700 1 -- 172.21.15.78:0/2593509420 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45bc074b80 con 0x7f45bc0c0830 2023-10-14T11:58:21.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.403+0000 7f5a1986a700 1 --2- 172.21.15.78:0/3948226941 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a140a7ba0 0x7f5a14072a70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5a140a58b0 tx=0x7f5a0400ac30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49c4217c2fd5741b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.403+0000 7f4140530700 1 -- 172.21.15.78:0/2181582495 shutdown_connections 2023-10-14T11:58:21.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.403+0000 7fd526ffd700 1 -- 172.21.15.78:0/4275551362 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd520011680 con 0x7fd5280706b0 2023-10-14T11:58:21.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.403+0000 7f5a18868700 1 -- 172.21.15.78:0/3948226941 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a040070c0 con 0x7f5a140a7ba0 2023-10-14T11:58:21.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.403+0000 7fd526ffd700 1 -- 172.21.15.78:0/4275551362 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5200117e0 con 0x7fd5280706b0 2023-10-14T11:58:21.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.403+0000 7fd526ffd700 1 -- 172.21.15.78:0/4275551362 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd520011b70 con 0x7fd5280706b0 2023-10-14T11:58:21.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.403+0000 7fd667fff700 1 -- 172.21.15.78:0/3126224691 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd6780083d0 con 0x7fd6800c2ed0 2023-10-14T11:58:21.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.403+0000 7efecd6c3700 1 Processor -- start 2023-10-14T11:58:21.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.403+0000 7f45a3fff700 1 --2- 172.21.15.78:0/2593509420 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f45a806af00 0x7f45a806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.403+0000 7efecd6c3700 1 -- start start 2023-10-14T11:58:21.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.403+0000 7f45a3fff700 1 -- 172.21.15.78:0/2593509420 <== mon.2 v2:172.21.15.195:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f45ac089260 con 0x7f45bc0c0830 2023-10-14T11:58:21.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.403+0000 7f4140530700 1 -- 172.21.15.78:0/2181582495 wait complete. 2023-10-14T11:58:21.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.403+0000 7fd667fff700 1 --2- 172.21.15.78:0/3126224691 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd66c06af00 0x7fd66c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.444 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.404+0000 7fd667fff700 1 -- 172.21.15.78:0/3126224691 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd67808b690 con 0x7fd6800c2ed0 2023-10-14T11:58:21.444 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.404+0000 7fd67effd700 1 --2- 172.21.15.78:0/3126224691 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd66c06af00 0x7fd66c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.444 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.403+0000 7f5a18868700 1 -- 172.21.15.78:0/3948226941 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a04007220 con 0x7f5a140a7ba0 2023-10-14T11:58:21.444 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.403+0000 7f5a18868700 1 -- 172.21.15.78:0/3948226941 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a04007590 con 0x7f5a140a7ba0 2023-10-14T11:58:21.445 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.404+0000 7f5a1bace700 1 -- 172.21.15.78:0/3948226941 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a140a7ba0 msgr2=0x7f5a14072a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.445 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.404+0000 7f5a1bace700 1 --2- 172.21.15.78:0/3948226941 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a140a7ba0 0x7f5a14072a70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5a140a58b0 tx=0x7f5a0400ac30 comp rx=0 tx=0).stop 2023-10-14T11:58:21.445 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.404+0000 7efecd6c3700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efec80c0830 0x7efec8072a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.404+0000 7efecd6c3700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7efec80c2880 0x7efec80c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.404+0000 7efecd6c3700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efec80c15d0 con 0x7efec80c59e0 2023-10-14T11:58:21.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.404+0000 7efecd6c3700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efec80c3f30 con 0x7efec80c2880 2023-10-14T11:58:21.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.404+0000 7efecd6c3700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efec80c4210 con 0x7efec80c0830 2023-10-14T11:58:21.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.404+0000 7efec77fe700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7efec80c59e0 0x7efec80a7930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49612/0 (socket says 172.21.15.78:49612) 2023-10-14T11:58:21.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.404+0000 7efec77fe700 1 -- 172.21.15.78:0/3810785037 learned_addr learned my addr 172.21.15.78:0/3810785037 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:21.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.404+0000 7efec6ffd700 1 --2- 172.21.15.78:0/3810785037 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efec80c0830 0x7efec8072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.404+0000 7efebe7fc700 1 --2- 172.21.15.78:0/3810785037 >> v2:172.21.15.78:3300/0 conn(0x7efec80c2880 0x7efec80c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.404+0000 7efec6ffd700 1 -- 172.21.15.78:0/3810785037 >> v2:172.21.15.78:3300/0 conn(0x7efec80c2880 msgr2=0x7efec80c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:21.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.404+0000 7efec6ffd700 1 --2- 172.21.15.78:0/3810785037 >> v2:172.21.15.78:3300/0 conn(0x7efec80c2880 0x7efec80c54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.404+0000 7efec6ffd700 1 -- 172.21.15.78:0/3810785037 >> v1:172.21.15.78:6789/0 conn(0x7efec80c59e0 legacy=0x7efec80a7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.404+0000 7efec6ffd700 1 -- 172.21.15.78:0/3810785037 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec80a8040 con 0x7efec80c0830 2023-10-14T11:58:21.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.405+0000 7fd67effd700 1 --2- 172.21.15.78:0/3126224691 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd66c06af00 0x7fd66c06d3c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd6800ba3a0 tx=0x7fd674006cf0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.405+0000 7efebe7fc700 1 --2- 172.21.15.78:0/3810785037 >> v2:172.21.15.78:3300/0 conn(0x7efec80c2880 0x7efec80c54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T11:58:21.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.404+0000 7fa9151c6700 1 Processor -- start 2023-10-14T11:58:21.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.405+0000 7fa9151c6700 1 -- start start 2023-10-14T11:58:21.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.405+0000 7f4140530700 1 Processor -- start 2023-10-14T11:58:21.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.405+0000 7efec6ffd700 1 --2- 172.21.15.78:0/3810785037 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efec80c0830 0x7efec8072a70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7efeb0009840 tx=0x7efeb0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c36e1988d4e793e8 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.405+0000 7f4140530700 1 -- start start 2023-10-14T11:58:21.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.406+0000 7fd52e8e6700 1 -- 172.21.15.78:0/4275551362 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5280706b0 msgr2=0x7fd5281548c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.406+0000 7fd52e8e6700 1 --2- 172.21.15.78:0/4275551362 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5280706b0 0x7fd5281548c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd52000c6a0 tx=0x7fd52000c9b0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.406+0000 7fa9151c6700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa9101523a0 0x7fa9100ab3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.406+0000 7fa9151c6700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa9100710e0 0x7fa910071550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.406+0000 7fa9151c6700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9100c0fe0 con 0x7fa91014ec80 2023-10-14T11:58:21.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.406+0000 7fa9151c6700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa910151790 con 0x7fa9100710e0 2023-10-14T11:58:21.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.406+0000 7fa9151c6700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9100c1280 con 0x7fa9101523a0 2023-10-14T11:58:21.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.406+0000 7fa90f59e700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa9100710e0 0x7fa910071550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.406+0000 7fa90f59e700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa9100710e0 0x7fa910071550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44746/0 (socket says 172.21.15.78:44746) 2023-10-14T11:58:21.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.406+0000 7fa90f59e700 1 -- 172.21.15.78:0/1320667561 learned_addr learned my addr 172.21.15.78:0/1320667561 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:21.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.406+0000 7fa907fff700 1 --2- 172.21.15.78:0/1320667561 >> v2:172.21.15.195:3300/0 conn(0x7fa9101523a0 0x7fa9100ab3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.407+0000 7f4140530700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f413806df80 0x7f41380bf930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.407+0000 7f4140530700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f41380a4ea0 0x7f41381febc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.407+0000 7f4140530700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41380c2ed0 0x7f41380bc620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.407+0000 7f4140530700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41381ff100 con 0x7f41380c2ed0 2023-10-14T11:58:21.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.407+0000 7f4140530700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41380abb80 con 0x7f413806df80 2023-10-14T11:58:21.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.407+0000 7f4140530700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41380bcb60 con 0x7f41380a4ea0 2023-10-14T11:58:21.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.406+0000 7fa90f59e700 1 -- 172.21.15.78:0/1320667561 >> v2:172.21.15.195:3300/0 conn(0x7fa9101523a0 msgr2=0x7fa9100ab3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:21.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.406+0000 7fa90f59e700 1 --2- 172.21.15.78:0/1320667561 >> v2:172.21.15.195:3300/0 conn(0x7fa9101523a0 0x7fa9100ab3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.406+0000 7fa90f59e700 1 -- 172.21.15.78:0/1320667561 >> v1:172.21.15.195:6789/0 conn(0x7fa91014ec80 legacy=0x7fa910151c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:21.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.406+0000 7fa90f59e700 1 -- 172.21.15.78:0/1320667561 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9100c05b0 con 0x7fa9100710e0 2023-10-14T11:58:21.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.407+0000 7fa90f59e700 1 --2- 172.21.15.78:0/1320667561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9100710e0 0x7fa910071550 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa910072fd0 tx=0x7fa90800d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=575ba930a8145b5f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.407+0000 7fd52e8e6700 1 -- 172.21.15.78:0/4275551362 shutdown_connections 2023-10-14T11:58:21.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.407+0000 7fd52e8e6700 1 --2- 172.21.15.78:0/4275551362 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5280706b0 0x7fd5281548c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.407+0000 7fd52e8e6700 1 --2- 172.21.15.78:0/4275551362 >> v2:172.21.15.78:3300/0 conn(0x7fd5280c0830 0x7fd528072a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.407+0000 7fd52e8e6700 1 -- 172.21.15.78:0/4275551362 >> 172.21.15.78:0/4275551362 conn(0x7fd52806aab0 msgr2=0x7fd52806aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:21.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.406+0000 7f45baffd700 1 --2- 172.21.15.78:0/2593509420 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f45a806af00 0x7f45a806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.407+0000 7f45baffd700 1 --2- 172.21.15.78:0/2593509420 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f45a806af00 0x7f45a806d3c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f45b0009c10 tx=0x7f45b0010000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.407+0000 7f413e2cc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f413806df80 0x7f41380bf930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.407+0000 7fd52e8e6700 1 -- 172.21.15.78:0/4275551362 shutdown_connections 2023-10-14T11:58:21.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.407+0000 7f413e2cc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f413806df80 0x7f41380bf930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.78:47488/0 (socket says 172.21.15.78:47488) 2023-10-14T11:58:21.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.407+0000 7f413e2cc700 1 -- 172.21.15.78:0/3689078248 learned_addr learned my addr 172.21.15.78:0/3689078248 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:21.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.407+0000 7f413dacb700 1 --2- 172.21.15.78:0/3689078248 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f41380a4ea0 0x7f41381febc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.407+0000 7f413eacd700 1 --2- 172.21.15.78:0/3689078248 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41380c2ed0 0x7f41380bc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.407+0000 7fd52e8e6700 1 -- 172.21.15.78:0/4275551362 wait complete. 2023-10-14T11:58:21.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.407+0000 7f5a1bace700 1 -- 172.21.15.78:0/3948226941 shutdown_connections 2023-10-14T11:58:21.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.407+0000 7f5a1bace700 1 --2- 172.21.15.78:0/3948226941 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a140a7ba0 0x7f5a14072a70 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.407+0000 7f5a1bace700 1 --2- 172.21.15.78:0/3948226941 >> v2:172.21.15.195:3300/0 conn(0x7f5a140c0830 0x7f5a140c54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.407+0000 7f5a1bace700 1 -- 172.21.15.78:0/3948226941 >> 172.21.15.78:0/3948226941 conn(0x7f5a1406aab0 msgr2=0x7f5a1406aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:21.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.407+0000 7fe13e164700 1 Processor -- start 2023-10-14T11:58:21.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.407+0000 7fe13e164700 1 -- start start 2023-10-14T11:58:21.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.407+0000 7f5a1bace700 1 -- 172.21.15.78:0/3948226941 shutdown_connections 2023-10-14T11:58:21.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.407+0000 7f5a1bace700 1 -- 172.21.15.78:0/3948226941 wait complete. 2023-10-14T11:58:21.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.407+0000 7f413dacb700 1 -- 172.21.15.78:0/3689078248 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f413806df80 msgr2=0x7f41380bf930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.407+0000 7f413dacb700 1 --2- 172.21.15.78:0/3689078248 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f413806df80 0x7f41380bf930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.407+0000 7f413dacb700 1 -- 172.21.15.78:0/3689078248 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41380c2ed0 msgr2=0x7f41380bc620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.407+0000 7f413dacb700 1 --2- 172.21.15.78:0/3689078248 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41380c2ed0 0x7f41380bc620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.407+0000 7f413dacb700 1 -- 172.21.15.78:0/3689078248 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4130009b30 con 0x7f41380a4ea0 2023-10-14T11:58:21.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.408+0000 7f413dacb700 1 --2- 172.21.15.78:0/3689078248 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f41380a4ea0 0x7f41381febc0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4128001380 tx=0x7f412800c7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.408+0000 7f412effd700 1 -- 172.21.15.78:0/3689078248 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41280054e0 con 0x7f41380a4ea0 2023-10-14T11:58:21.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.408+0000 7efec67fc700 1 -- 172.21.15.78:0/3810785037 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb0011830 con 0x7efec80c0830 2023-10-14T11:58:21.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.408+0000 7efec67fc700 1 -- 172.21.15.78:0/3810785037 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeb0011990 con 0x7efec80c0830 2023-10-14T11:58:21.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.408+0000 7efec67fc700 1 -- 172.21.15.78:0/3810785037 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb0011d20 con 0x7efec80c0830 2023-10-14T11:58:21.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.408+0000 7fe13e164700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe1380afad0 0x7fe1380c66e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.408+0000 7fe13e164700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe138155ab0 con 0x7fe13806b180 2023-10-14T11:58:21.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.408+0000 7fe13e164700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1380c6250 con 0x7fe1380c4f40 2023-10-14T11:58:21.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.408+0000 7fe13e164700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe138156680 con 0x7fe1380afad0 2023-10-14T11:58:21.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.408+0000 7f412effd700 1 -- 172.21.15.78:0/3689078248 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f412800ce70 con 0x7f41380a4ea0 2023-10-14T11:58:21.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.408+0000 7f4140530700 1 -- 172.21.15.78:0/3689078248 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41380ab8a0 con 0x7f41380a4ea0 2023-10-14T11:58:21.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.408+0000 7f4140530700 1 -- 172.21.15.78:0/3689078248 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41380a8090 con 0x7f41380a4ea0 2023-10-14T11:58:21.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.408+0000 7f412effd700 1 -- 172.21.15.78:0/3689078248 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41280038a0 con 0x7f41380a4ea0 2023-10-14T11:58:21.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.409+0000 7fd52e8e6700 1 Processor -- start 2023-10-14T11:58:21.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.409+0000 7efecd6c3700 1 -- 172.21.15.78:0/3810785037 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efec80c0830 msgr2=0x7efec8072a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.409+0000 7efecd6c3700 1 --2- 172.21.15.78:0/3810785037 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efec80c0830 0x7efec8072a70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7efeb0009840 tx=0x7efeb0009b50 comp rx=0 tx=0).stop 2023-10-14T11:58:21.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.409+0000 7efecd6c3700 1 -- 172.21.15.78:0/3810785037 shutdown_connections 2023-10-14T11:58:21.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.409+0000 7efecd6c3700 1 --2- 172.21.15.78:0/3810785037 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efec80c0830 0x7efec8072a70 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7efeb0009840 tx=0x7efeb0009b50 comp rx=0 tx=0).stop 2023-10-14T11:58:21.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.409+0000 7efecd6c3700 1 --2- 172.21.15.78:0/3810785037 >> v2:172.21.15.78:3300/0 conn(0x7efec80c2880 0x7efec80c54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.409+0000 7efecd6c3700 1 -- 172.21.15.78:0/3810785037 >> 172.21.15.78:0/3810785037 conn(0x7efec806aab0 msgr2=0x7efec806aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:21.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.409+0000 7fe136ffd700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fe1380c4f40 0x7fe138155f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.78:39434/0 (socket says 172.21.15.78:39434) 2023-10-14T11:58:21.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.409+0000 7fe136ffd700 1 -- 172.21.15.78:0/2058995990 learned_addr learned my addr 172.21.15.78:0/2058995990 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:21.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.409+0000 7fe1377fe700 1 --2- 172.21.15.78:0/2058995990 >> v2:172.21.15.78:3300/0 conn(0x7fe1380afad0 0x7fe1380c66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.409+0000 7fe1377fe700 1 -- 172.21.15.78:0/2058995990 >> v1:172.21.15.195:6789/0 conn(0x7fe1380c4f40 legacy=0x7fe138155f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.409+0000 7fe1377fe700 1 -- 172.21.15.78:0/2058995990 >> v1:172.21.15.78:6789/0 conn(0x7fe13806b180 legacy=0x7fe13806b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:21.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.409+0000 7fe1377fe700 1 -- 172.21.15.78:0/2058995990 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1381525a0 con 0x7fe1380afad0 2023-10-14T11:58:21.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.409+0000 7fe1377fe700 1 --2- 172.21.15.78:0/2058995990 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1380afad0 0x7fe1380c66e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe128009840 tx=0x7fe128009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fecc8fac54d2a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.410+0000 7fd52e8e6700 1 -- start start 2023-10-14T11:58:21.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.409+0000 7f5a1bace700 1 Processor -- start 2023-10-14T11:58:21.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.410+0000 7f4140530700 1 -- 172.21.15.78:0/3689078248 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41200052f0 con 0x7f41380a4ea0 2023-10-14T11:58:21.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.410+0000 7f5a1bace700 1 -- start start 2023-10-14T11:58:21.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.410+0000 7f412effd700 1 -- 172.21.15.78:0/3689078248 <== mon.2 v2:172.21.15.195:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4128003a00 con 0x7f41380a4ea0 2023-10-14T11:58:21.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.411+0000 7f412effd700 1 --2- 172.21.15.78:0/3689078248 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f411c06af00 0x7f411c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.411+0000 7f412effd700 1 -- 172.21.15.78:0/3689078248 <== mon.2 v2:172.21.15.195:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4128088710 con 0x7f41380a4ea0 2023-10-14T11:58:21.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.411+0000 7fa90e59c700 1 -- 172.21.15.78:0/1320667561 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa908010070 con 0x7fa9100710e0 2023-10-14T11:58:21.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.411+0000 7fd52e8e6700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd5280706b0 0x7fd5280bf260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.411+0000 7fd52e8e6700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5280a7ba0 0x7fd5280aa320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.411+0000 7fd52e8e6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5280c0830 0x7fd5280bd490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.411+0000 7fd52e8e6700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5280c65c0 con 0x7fd5280c0830 2023-10-14T11:58:21.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.411+0000 7fd52e8e6700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5280a9e90 con 0x7fd5280706b0 2023-10-14T11:58:21.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.411+0000 7fd52e8e6700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd528149880 con 0x7fd5280a7ba0 2023-10-14T11:58:21.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.411+0000 7fd667fff700 1 -- 172.21.15.78:0/3126224691 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd6800ba680 con 0x7fd6800c2ed0 2023-10-14T11:58:21.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.411+0000 7f5a1bace700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5a140a7ba0 0x7f5a1414c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.411+0000 7f5a1bace700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a140c0830 0x7f5a141f1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.411+0000 7f5a1bace700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a140c59e0 0x7f5a140aab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.411+0000 7f5a1bace700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a140a50e0 con 0x7f5a140c0830 2023-10-14T11:58:21.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.411+0000 7f5a1bace700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a140aa860 con 0x7f5a140c59e0 2023-10-14T11:58:21.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.411+0000 7f5a1bace700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1414c580 con 0x7f5a140a7ba0 2023-10-14T11:58:21.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.411+0000 7fe1367fc700 1 -- 172.21.15.78:0/2058995990 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe128004230 con 0x7fe1380afad0 2023-10-14T11:58:21.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.411+0000 7f413e2cc700 1 --2- 172.21.15.78:0/3689078248 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f411c06af00 0x7f411c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.411+0000 7fd5277fe700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5280a7ba0 0x7fd5280aa320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.411+0000 7fd5277fe700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5280a7ba0 0x7fd5280aa320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.78:44436/0 (socket says 172.21.15.78:44436) 2023-10-14T11:58:21.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.411+0000 7fd5277fe700 1 -- 172.21.15.78:0/908389582 learned_addr learned my addr 172.21.15.78:0/908389582 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:21.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.410+0000 7efecd6c3700 1 -- 172.21.15.78:0/3810785037 shutdown_connections 2023-10-14T11:58:21.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.411+0000 7efecd6c3700 1 -- 172.21.15.78:0/3810785037 wait complete. 2023-10-14T11:58:21.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.411+0000 7f5a1986a700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5a140a7ba0 0x7f5a1414c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.411+0000 7f5a1a06b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a140c59e0 0x7f5a140aab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.411+0000 7f5a1986a700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5a140a7ba0 0x7f5a1414c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.78:44442/0 (socket says 172.21.15.78:44442) 2023-10-14T11:58:21.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.411+0000 7f5a1986a700 1 -- 172.21.15.78:0/1784546362 learned_addr learned my addr 172.21.15.78:0/1784546362 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:21.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.411+0000 7f413e2cc700 1 --2- 172.21.15.78:0/3689078248 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f411c06af00 0x7f411c06d3c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4134006fd0 tx=0x7f4134008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.412+0000 7fd5277fe700 1 -- 172.21.15.78:0/908389582 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd5280706b0 msgr2=0x7fd5280bf260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.412+0000 7fd5277fe700 1 --2- 172.21.15.78:0/908389582 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd5280706b0 0x7fd5280bf260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.412+0000 7fd5277fe700 1 -- 172.21.15.78:0/908389582 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5280c0830 msgr2=0x7fd5280bd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.412+0000 7fd5277fe700 1 --2- 172.21.15.78:0/908389582 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5280c0830 0x7fd5280bd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.412+0000 7fd5277fe700 1 -- 172.21.15.78:0/908389582 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd52000c300 con 0x7fd5280a7ba0 2023-10-14T11:58:21.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.412+0000 7f5a1a06b700 1 -- 172.21.15.78:0/1784546362 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5a140a7ba0 msgr2=0x7f5a1414c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.412+0000 7f5a1a06b700 1 --2- 172.21.15.78:0/1784546362 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5a140a7ba0 0x7f5a1414c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.412+0000 7f5a1a06b700 1 -- 172.21.15.78:0/1784546362 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a140c0830 msgr2=0x7f5a141f1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.412+0000 7f5a1a06b700 1 --2- 172.21.15.78:0/1784546362 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a140c0830 0x7f5a141f1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.412+0000 7f5a1a06b700 1 -- 172.21.15.78:0/1784546362 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a0400a5b0 con 0x7f5a140c59e0 2023-10-14T11:58:21.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.412+0000 7fd5277fe700 1 --2- 172.21.15.78:0/908389582 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5280a7ba0 0x7fd5280aa320 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd51800b4a0 tx=0x7fd51800b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.412+0000 7f5a1a06b700 1 --2- 172.21.15.78:0/1784546362 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a140c59e0 0x7f5a140aab00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c00c700 tx=0x7f5a0c00ca10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.412+0000 7f5a0a7fc700 1 -- 172.21.15.78:0/1784546362 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c011790 con 0x7f5a140c59e0 2023-10-14T11:58:21.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.412+0000 7f5a0a7fc700 1 -- 172.21.15.78:0/1784546362 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a0c0118f0 con 0x7f5a140c59e0 2023-10-14T11:58:21.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.412+0000 7f5a0a7fc700 1 -- 172.21.15.78:0/1784546362 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c011ca0 con 0x7f5a140c59e0 2023-10-14T11:58:21.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.412+0000 7f5a1bace700 1 -- 172.21.15.78:0/1784546362 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a14148e70 con 0x7f5a140c59e0 2023-10-14T11:58:21.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.412+0000 7f45a3fff700 1 -- 172.21.15.78:0/2593509420 <== mon.2 v2:172.21.15.195:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f45ac054280 con 0x7f45bc0c0830 2023-10-14T11:58:21.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.412+0000 7f5a1bace700 1 -- 172.21.15.78:0/1784546362 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a14149430 con 0x7f5a140c59e0 2023-10-14T11:58:21.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.412+0000 7fe1367fc700 1 -- 172.21.15.78:0/2058995990 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe128004390 con 0x7fe1380afad0 2023-10-14T11:58:21.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.412+0000 7fe1367fc700 1 -- 172.21.15.78:0/2058995990 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1280036a0 con 0x7fe1380afad0 2023-10-14T11:58:21.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.413+0000 7fe13e164700 1 -- 172.21.15.78:0/2058995990 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1380afad0 msgr2=0x7fe1380c66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.413+0000 7fe13e164700 1 --2- 172.21.15.78:0/2058995990 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1380afad0 0x7fe1380c66e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe128009840 tx=0x7fe128009b50 comp rx=0 tx=0).stop 2023-10-14T11:58:21.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.411+0000 7fa90e59c700 1 -- 172.21.15.78:0/1320667561 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa90800b2e0 con 0x7fa9100710e0 2023-10-14T11:58:21.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.411+0000 7fa90e59c700 1 -- 172.21.15.78:0/1320667561 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90800dcd0 con 0x7fa9100710e0 2023-10-14T11:58:21.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.412+0000 7fa9151c6700 1 -- 172.21.15.78:0/1320667561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9100710e0 msgr2=0x7fa910071550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.412+0000 7fa9151c6700 1 --2- 172.21.15.78:0/1320667561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9100710e0 0x7fa910071550 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa910072fd0 tx=0x7fa90800d430 comp rx=0 tx=0).stop 2023-10-14T11:58:21.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.412+0000 7fa9151c6700 1 -- 172.21.15.78:0/1320667561 shutdown_connections 2023-10-14T11:58:21.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.412+0000 7fa9151c6700 1 --2- 172.21.15.78:0/1320667561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9100710e0 0x7fa910071550 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.412+0000 7fa9151c6700 1 --2- 172.21.15.78:0/1320667561 >> v2:172.21.15.195:3300/0 conn(0x7fa9101523a0 0x7fa9100ab3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.412+0000 7fa9151c6700 1 -- 172.21.15.78:0/1320667561 >> 172.21.15.78:0/1320667561 conn(0x7fa9100c6310 msgr2=0x7fa9100c66f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:21.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.412+0000 7fa9151c6700 1 -- 172.21.15.78:0/1320667561 shutdown_connections 2023-10-14T11:58:21.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.412+0000 7fa9151c6700 1 -- 172.21.15.78:0/1320667561 wait complete. 2023-10-14T11:58:21.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.414+0000 7fe13e164700 1 -- 172.21.15.78:0/2058995990 shutdown_connections 2023-10-14T11:58:21.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.414+0000 7fe13e164700 1 --2- 172.21.15.78:0/2058995990 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1380afad0 0x7fe1380c66e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.414+0000 7fe13e164700 1 -- 172.21.15.78:0/2058995990 >> 172.21.15.78:0/2058995990 conn(0x7fe1380c6e20 msgr2=0x7fe1380c7200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:21.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.414+0000 7fa9151c6700 1 Processor -- start 2023-10-14T11:58:21.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.414+0000 7fa9151c6700 1 -- start start 2023-10-14T11:58:21.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.413+0000 7efecd6c3700 1 Processor -- start 2023-10-14T11:58:21.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.413+0000 7efecd6c3700 1 -- start start 2023-10-14T11:58:21.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.414+0000 7efecd6c3700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efec80c2880 0x7efec8153170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.414+0000 7efecd6c3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efec80c59e0 0x7efec80b8340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.414+0000 7efecd6c3700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efec80bc780 0x7efec80b8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.414+0000 7efecd6c3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec81fe5f0 con 0x7efec80c59e0 2023-10-14T11:58:21.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.414+0000 7efecd6c3700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec81fcea0 con 0x7efec80bc780 2023-10-14T11:58:21.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.414+0000 7efecd6c3700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efec80b8dc0 con 0x7efec80c2880 2023-10-14T11:58:21.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.414+0000 7f5a0a7fc700 1 -- 172.21.15.78:0/1784546362 <== mon.1 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f5a0c01c460 con 0x7f5a140c59e0 2023-10-14T11:58:21.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.414+0000 7f5a1bace700 1 -- 172.21.15.78:0/1784546362 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a141495c0 con 0x7f5a140c59e0 2023-10-14T11:58:21.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.415+0000 7f5a0a7fc700 1 --2- 172.21.15.78:0/1784546362 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f5a0006b370 0x7f5a0006d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.415+0000 7f5a1986a700 1 --2- 172.21.15.78:0/1784546362 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f5a0006b370 0x7f5a0006d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.415+0000 7f5a0a7fc700 1 -- 172.21.15.78:0/1784546362 <== mon.1 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5a0c089530 con 0x7f5a140c59e0 2023-10-14T11:58:21.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.415+0000 7fa9151c6700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9100710e0 0x7fa910146740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.415+0000 7fa9151c6700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa91014ec80 0x7fa91014a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.415+0000 7fa9151c6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9101523a0 0x7fa910141030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.415+0000 7fa9151c6700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9101e57e0 con 0x7fa9101523a0 2023-10-14T11:58:21.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.415+0000 7fa9151c6700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa910147e50 con 0x7fa9100710e0 2023-10-14T11:58:21.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.415+0000 7fa9151c6700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa910146d90 con 0x7fa91014ec80 2023-10-14T11:58:21.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.415+0000 7efec6ffd700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efec80c2880 0x7efec8153170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.415+0000 7efec6ffd700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efec80c2880 0x7efec8153170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.78:44450/0 (socket says 172.21.15.78:44450) 2023-10-14T11:58:21.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.415+0000 7efec6ffd700 1 -- 172.21.15.78:0/3599129615 learned_addr learned my addr 172.21.15.78:0/3599129615 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:21.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.415+0000 7efec77fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efec80bc780 0x7efec80b8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.415+0000 7f5a1986a700 1 --2- 172.21.15.78:0/1784546362 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f5a0006b370 0x7f5a0006d830 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5a0400b010 tx=0x7f5a0400f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.415+0000 7fa907fff700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa91014ec80 0x7fa91014a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.415+0000 7fa907fff700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa91014ec80 0x7fa91014a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.78:44454/0 (socket says 172.21.15.78:44454) 2023-10-14T11:58:21.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.415+0000 7fa907fff700 1 -- 172.21.15.78:0/2928006316 learned_addr learned my addr 172.21.15.78:0/2928006316 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:21.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.415+0000 7fa90ed9d700 1 --2- 172.21.15.78:0/2928006316 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9100710e0 0x7fa910146740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.415+0000 7efec6ffd700 1 -- 172.21.15.78:0/3599129615 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efec80bc780 msgr2=0x7efec80b8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.415+0000 7fd524ff9700 1 -- 172.21.15.78:0/908389582 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd518011830 con 0x7fd5280a7ba0 2023-10-14T11:58:21.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.416+0000 7fd524ff9700 1 -- 172.21.15.78:0/908389582 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd518011990 con 0x7fd5280a7ba0 2023-10-14T11:58:21.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.416+0000 7fd524ff9700 1 -- 172.21.15.78:0/908389582 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd518011d20 con 0x7fd5280a7ba0 2023-10-14T11:58:21.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.416+0000 7fd52e8e6700 1 -- 172.21.15.78:0/908389582 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5280b8cc0 con 0x7fd5280a7ba0 2023-10-14T11:58:21.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.416+0000 7fa90f59e700 1 --2- 172.21.15.78:0/2928006316 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9101523a0 0x7fa910141030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.416+0000 7efec6ffd700 1 --2- 172.21.15.78:0/3599129615 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efec80bc780 0x7efec80b8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.416+0000 7efec6ffd700 1 -- 172.21.15.78:0/3599129615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efec80c59e0 msgr2=0x7efec80b8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.416+0000 7efec6ffd700 1 --2- 172.21.15.78:0/3599129615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efec80c59e0 0x7efec80b8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.416+0000 7efec6ffd700 1 -- 172.21.15.78:0/3599129615 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb80094a0 con 0x7efec80c2880 2023-10-14T11:58:21.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.416+0000 7efec77fe700 1 --2- 172.21.15.78:0/3599129615 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efec80bc780 0x7efec80b8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T11:58:21.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.416+0000 7fd52e8e6700 1 -- 172.21.15.78:0/908389582 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5280a9b20 con 0x7fd5280a7ba0 2023-10-14T11:58:21.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.416+0000 7fa90ed9d700 1 -- 172.21.15.78:0/2928006316 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa91014ec80 msgr2=0x7fa91014a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.416+0000 7fa90ed9d700 1 --2- 172.21.15.78:0/2928006316 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa91014ec80 0x7fa91014a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.416+0000 7fa90ed9d700 1 -- 172.21.15.78:0/2928006316 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9101523a0 msgr2=0x7fa910141030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.416+0000 7fa90ed9d700 1 --2- 172.21.15.78:0/2928006316 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9101523a0 0x7fa910141030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.416+0000 7fa90ed9d700 1 -- 172.21.15.78:0/2928006316 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa90800bd40 con 0x7fa9100710e0 2023-10-14T11:58:21.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.416+0000 7fa90ed9d700 1 --2- 172.21.15.78:0/2928006316 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9100710e0 0x7fa910146740 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa8f800c440 tx=0x7fa8f800c750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.416+0000 7efec6ffd700 1 --2- 172.21.15.78:0/3599129615 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efec80c2880 0x7efec8153170 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efeb00090b0 tx=0x7efeb0004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.416+0000 7efebffff700 1 -- 172.21.15.78:0/3599129615 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb00045d0 con 0x7efec80c2880 2023-10-14T11:58:21.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.416+0000 7efebffff700 1 -- 172.21.15.78:0/3599129615 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeb001f070 con 0x7efec80c2880 2023-10-14T11:58:21.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.416+0000 7efebffff700 1 -- 172.21.15.78:0/3599129615 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb00038a0 con 0x7efec80c2880 2023-10-14T11:58:21.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.416+0000 7efecd6c3700 1 -- 172.21.15.78:0/3599129615 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeb00094a0 con 0x7efec80c2880 2023-10-14T11:58:21.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.416+0000 7efecd6c3700 1 -- 172.21.15.78:0/3599129615 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efec81fe840 con 0x7efec80c2880 2023-10-14T11:58:21.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.417+0000 7fd52e8e6700 1 -- 172.21.15.78:0/908389582 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5280c1150 con 0x7fd5280a7ba0 2023-10-14T11:58:21.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.417+0000 7fd524ff9700 1 -- 172.21.15.78:0/908389582 <== mon.2 v2:172.21.15.195:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd51801a400 con 0x7fd5280a7ba0 2023-10-14T11:58:21.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.418+0000 7f412effd700 1 -- 172.21.15.78:0/3689078248 <== mon.2 v2:172.21.15.195:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4128053940 con 0x7f41380a4ea0 2023-10-14T11:58:21.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.418+0000 7efecd6c3700 1 -- 172.21.15.78:0/3599129615 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efec8074b80 con 0x7efec80c2880 2023-10-14T11:58:21.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.418+0000 7fd524ff9700 1 --2- 172.21.15.78:0/908389582 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd51406af00 0x7fd51406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.418+0000 7fd524ff9700 1 -- 172.21.15.78:0/908389582 <== mon.2 v2:172.21.15.195:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd5180895b0 con 0x7fd5280a7ba0 2023-10-14T11:58:21.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.419+0000 7efebffff700 1 -- 172.21.15.78:0/3599129615 <== mon.2 v2:172.21.15.195:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7efeb0003a00 con 0x7efec80c2880 2023-10-14T11:58:21.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.419+0000 7fe13e164700 1 -- 172.21.15.78:0/2058995990 shutdown_connections 2023-10-14T11:58:21.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.419+0000 7fe13e164700 1 -- 172.21.15.78:0/2058995990 wait complete. 2023-10-14T11:58:21.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.419+0000 7efebffff700 1 --2- 172.21.15.78:0/3599129615 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7efea806af00 0x7efea806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.419+0000 7efebe7fc700 1 --2- 172.21.15.78:0/3599129615 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7efea806af00 0x7efea806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.419+0000 7efebffff700 1 -- 172.21.15.78:0/3599129615 <== mon.2 v2:172.21.15.195:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efeb0021070 con 0x7efec80c2880 2023-10-14T11:58:21.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.420+0000 7efebe7fc700 1 --2- 172.21.15.78:0/3599129615 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7efea806af00 0x7efea806d3c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7efeb8006f10 tx=0x7efeb8006e60 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.420+0000 7fa9077fe700 1 -- 172.21.15.78:0/2928006316 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f80052d0 con 0x7fa9100710e0 2023-10-14T11:58:21.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.418+0000 7fd527fff700 1 --2- 172.21.15.78:0/908389582 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd51406af00 0x7fd51406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.418+0000 7fd527fff700 1 --2- 172.21.15.78:0/908389582 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd51406af00 0x7fd51406d3c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd51c000f60 tx=0x7fd51c007b10 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.420+0000 7fa9077fe700 1 -- 172.21.15.78:0/2928006316 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8f8005430 con 0x7fa9100710e0 2023-10-14T11:58:21.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.420+0000 7fa9151c6700 1 -- 172.21.15.78:0/2928006316 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9101e6790 con 0x7fa9100710e0 2023-10-14T11:58:21.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.420+0000 7fa9077fe700 1 -- 172.21.15.78:0/2928006316 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f8003910 con 0x7fa9100710e0 2023-10-14T11:58:21.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.420+0000 7fa9151c6700 1 -- 172.21.15.78:0/2928006316 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9101e5cb0 con 0x7fa9100710e0 2023-10-14T11:58:21.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.420+0000 7fe13e164700 1 Processor -- start 2023-10-14T11:58:21.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.420+0000 7fe13e164700 1 -- start start 2023-10-14T11:58:21.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.422+0000 7fe13e164700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe13806b180 0x7fe1380c13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.422+0000 7fe13e164700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe1380afad0 0x7fe138150e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.422+0000 7fe13e164700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1380c4f40 0x7fe1380c09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.422+0000 7fe13e164700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1380be300 con 0x7fe1380c4f40 2023-10-14T11:58:21.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.422+0000 7fe13e164700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe138151380 con 0x7fe13806b180 2023-10-14T11:58:21.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.422+0000 7fe13e164700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1380c32f0 con 0x7fe1380afad0 2023-10-14T11:58:21.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.422+0000 7fe1377fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe13806b180 0x7fe1380c13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.422+0000 7fe1377fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe13806b180 0x7fe1380c13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.78:47532/0 (socket says 172.21.15.78:47532) 2023-10-14T11:58:21.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.422+0000 7fe1377fe700 1 -- 172.21.15.78:0/2384937077 learned_addr learned my addr 172.21.15.78:0/2384937077 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:21.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.422+0000 7fe136ffd700 1 --2- 172.21.15.78:0/2384937077 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe1380afad0 0x7fe138150e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.422+0000 7f5a0a7fc700 1 -- 172.21.15.78:0/1784546362 <== mon.1 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5a0c0545d0 con 0x7f5a140c59e0 2023-10-14T11:58:21.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.422+0000 7fa9077fe700 1 -- 172.21.15.78:0/2928006316 <== mon.1 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa8f8003a70 con 0x7fa9100710e0 2023-10-14T11:58:21.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.422+0000 7fe1377fe700 1 -- 172.21.15.78:0/2384937077 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe1380afad0 msgr2=0x7fe138150e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.422+0000 7fe1377fe700 1 --2- 172.21.15.78:0/2384937077 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe1380afad0 0x7fe138150e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.422+0000 7fe1377fe700 1 -- 172.21.15.78:0/2384937077 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1380c4f40 msgr2=0x7fe1380c09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.422+0000 7fe1377fe700 1 --2- 172.21.15.78:0/2384937077 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1380c4f40 0x7fe1380c09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.422+0000 7fe1377fe700 1 -- 172.21.15.78:0/2384937077 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1280094a0 con 0x7fe13806b180 2023-10-14T11:58:21.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.423+0000 7fa9077fe700 1 --2- 172.21.15.78:0/2928006316 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa8f006af00 0x7fa8f006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.422+0000 7fe1377fe700 1 --2- 172.21.15.78:0/2384937077 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe13806b180 0x7fe1380c13e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe1280090b0 tx=0x7fe128004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.423+0000 7fa9077fe700 1 -- 172.21.15.78:0/2928006316 <== mon.1 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa8f8088810 con 0x7fa9100710e0 2023-10-14T11:58:21.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.423+0000 7fe11ffff700 1 -- 172.21.15.78:0/2384937077 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe128011630 con 0x7fe13806b180 2023-10-14T11:58:21.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.423+0000 7fd524ff9700 1 -- 172.21.15.78:0/908389582 <== mon.2 v2:172.21.15.195:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd518054670 con 0x7fd5280a7ba0 2023-10-14T11:58:21.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.424+0000 7fa907fff700 1 --2- 172.21.15.78:0/2928006316 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa8f006af00 0x7fa8f006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.423+0000 7fe11ffff700 1 -- 172.21.15.78:0/2384937077 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe12801e070 con 0x7fe13806b180 2023-10-14T11:58:21.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.423+0000 7fe11ffff700 1 -- 172.21.15.78:0/2384937077 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1280119f0 con 0x7fe13806b180 2023-10-14T11:58:21.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.424+0000 7fe13e164700 1 -- 172.21.15.78:0/2384937077 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1380c4360 con 0x7fe13806b180 2023-10-14T11:58:21.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.424+0000 7fe13e164700 1 -- 172.21.15.78:0/2384937077 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1380bdf20 con 0x7fe13806b180 2023-10-14T11:58:21.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.424+0000 7fa9151c6700 1 -- 172.21.15.78:0/2928006316 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8f40052f0 con 0x7fa9100710e0 2023-10-14T11:58:21.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.425+0000 7fa907fff700 1 --2- 172.21.15.78:0/2928006316 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa8f006af00 0x7fa8f006d3c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa91014b660 tx=0x7fa900009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.425+0000 7fe13e164700 1 -- 172.21.15.78:0/2384937077 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1380be0e0 con 0x7fe13806b180 2023-10-14T11:58:21.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.427+0000 7efebffff700 1 -- 172.21.15.78:0/3599129615 <== mon.2 v2:172.21.15.195:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7efeb0055e60 con 0x7efec80c2880 2023-10-14T11:58:21.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.430+0000 7fa9077fe700 1 -- 172.21.15.78:0/2928006316 <== mon.1 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa8f8017080 con 0x7fa9100710e0 2023-10-14T11:58:21.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.429+0000 7fe11ffff700 1 -- 172.21.15.78:0/2384937077 <== mon.1 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe128011b50 con 0x7fe13806b180 2023-10-14T11:58:21.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.429+0000 7fe11ffff700 1 --2- 172.21.15.78:0/2384937077 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe12406af80 0x7fe12406d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:21.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.429+0000 7fe11ffff700 1 -- 172.21.15.78:0/2384937077 <== mon.1 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe12808a930 con 0x7fe13806b180 2023-10-14T11:58:21.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.429+0000 7fe136ffd700 1 --2- 172.21.15.78:0/2384937077 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe12406af80 0x7fe12406d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:21.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.430+0000 7fe136ffd700 1 --2- 172.21.15.78:0/2384937077 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe12406af80 0x7fe12406d440 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe1380af2c0 tx=0x7fe12c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:21.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.434+0000 7fe11ffff700 1 -- 172.21.15.78:0/2384937077 <== mon.1 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe128055ad0 con 0x7fe13806b180 2023-10-14T11:58:21.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.806+0000 7f4140530700 1 -- 172.21.15.78:0/3689078248 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f4120005870 con 0x7f41380a4ea0 2023-10-14T11:58:21.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.806+0000 7f412effd700 1 -- 172.21.15.78:0/3689078248 <== mon.2 v2:172.21.15.195:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4128017090 con 0x7f41380a4ea0 2023-10-14T11:58:21.811 INFO:teuthology.orchestra.run.smithi078.stdout:55834574851 2023-10-14T11:58:21.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.810+0000 7f4140530700 1 -- 172.21.15.78:0/3689078248 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f411c06af00 msgr2=0x7f411c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.810+0000 7f4140530700 1 --2- 172.21.15.78:0/3689078248 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f411c06af00 0x7f411c06d3c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4134006fd0 tx=0x7f4134008040 comp rx=0 tx=0).stop 2023-10-14T11:58:21.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.810+0000 7f4140530700 1 -- 172.21.15.78:0/3689078248 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f41380a4ea0 msgr2=0x7f41381febc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.810+0000 7f4140530700 1 --2- 172.21.15.78:0/3689078248 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f41380a4ea0 0x7f41381febc0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4128001380 tx=0x7f412800c7e0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.810+0000 7f4140530700 1 -- 172.21.15.78:0/3689078248 shutdown_connections 2023-10-14T11:58:21.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.810+0000 7f4140530700 1 --2- 172.21.15.78:0/3689078248 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41380c2ed0 0x7f41380bc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.810+0000 7f4140530700 1 --2- 172.21.15.78:0/3689078248 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f411c06af00 0x7f411c06d3c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.810+0000 7f4140530700 1 --2- 172.21.15.78:0/3689078248 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f413806df80 0x7f41380bf930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.810+0000 7f4140530700 1 --2- 172.21.15.78:0/3689078248 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f41380a4ea0 0x7f41381febc0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.811+0000 7f4140530700 1 -- 172.21.15.78:0/3689078248 >> 172.21.15.78:0/3689078248 conn(0x7f413806aab0 msgr2=0x7f4138071920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:21.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.811+0000 7f4140530700 1 -- 172.21.15.78:0/3689078248 shutdown_connections 2023-10-14T11:58:21.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.811+0000 7f4140530700 1 -- 172.21.15.78:0/3689078248 wait complete. 2023-10-14T11:58:21.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.817+0000 7f45c1aa4700 1 -- 172.21.15.78:0/2593509420 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f45bc0abe30 con 0x7f45bc0c0830 2023-10-14T11:58:21.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.817+0000 7fd685f22700 1 -- 172.21.15.78:0/3126224691 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fd6800ba680 con 0x7fd6800c2ed0 2023-10-14T11:58:21.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.817+0000 7fd667fff700 1 -- 172.21.15.78:0/3126224691 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd6800ba680 con 0x7fd6800c2ed0 2023-10-14T11:58:21.820 INFO:teuthology.orchestra.run.smithi078.stdout:55834574852 2023-10-14T11:58:21.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.818+0000 7f45a3fff700 1 -- 172.21.15.78:0/2593509420 <== mon.2 v2:172.21.15.195:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f45ac0092a0 con 0x7f45bc0c0830 2023-10-14T11:58:21.823 INFO:teuthology.orchestra.run.smithi078.stdout:55834574851 2023-10-14T11:58:21.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.822+0000 7f45c1aa4700 1 -- 172.21.15.78:0/2593509420 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f45a806af00 msgr2=0x7f45a806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.822+0000 7f45c1aa4700 1 --2- 172.21.15.78:0/2593509420 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f45a806af00 0x7f45a806d3c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f45b0009c10 tx=0x7f45b0010000 comp rx=0 tx=0).stop 2023-10-14T11:58:21.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.822+0000 7f45c1aa4700 1 -- 172.21.15.78:0/2593509420 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f45bc0c0830 msgr2=0x7f45bc1ed3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.822+0000 7f45c1aa4700 1 --2- 172.21.15.78:0/2593509420 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f45bc0c0830 0x7f45bc1ed3f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f45ac00b4a0 tx=0x7f45ac00b860 comp rx=0 tx=0).stop 2023-10-14T11:58:21.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.822+0000 7fd685f22700 1 -- 172.21.15.78:0/3126224691 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd66c06af00 msgr2=0x7fd66c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.822+0000 7fd685f22700 1 --2- 172.21.15.78:0/3126224691 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd66c06af00 0x7fd66c06d3c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd6800ba3a0 tx=0x7fd674006cf0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.822+0000 7fd685f22700 1 -- 172.21.15.78:0/3126224691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6800c2ed0 msgr2=0x7fd6800b9930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.822+0000 7fd685f22700 1 --2- 172.21.15.78:0/3126224691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6800c2ed0 0x7fd6800b9930 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd67800d8f0 tx=0x7fd678004ad0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.822+0000 7f45c1aa4700 1 -- 172.21.15.78:0/2593509420 shutdown_connections 2023-10-14T11:58:21.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.822+0000 7f45c1aa4700 1 --2- 172.21.15.78:0/2593509420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45bc0a9a50 0x7f45bc0aab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.822+0000 7f45c1aa4700 1 --2- 172.21.15.78:0/2593509420 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f45bc0c0830 0x7f45bc1ed3f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.822+0000 7f45c1aa4700 1 --2- 172.21.15.78:0/2593509420 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f45a806af00 0x7f45a806d3c0 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f45b0009c10 tx=0x7f45b0010000 comp rx=0 tx=0).stop 2023-10-14T11:58:21.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.822+0000 7f45c1aa4700 1 --2- 172.21.15.78:0/2593509420 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f45bc0c59e0 0x7f45bc0aa650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.822+0000 7f45c1aa4700 1 -- 172.21.15.78:0/2593509420 >> 172.21.15.78:0/2593509420 conn(0x7f45bc06aab0 msgr2=0x7f45bc0a63a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:21.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.822+0000 7fd685f22700 1 -- 172.21.15.78:0/3126224691 shutdown_connections 2023-10-14T11:58:21.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.822+0000 7fd685f22700 1 --2- 172.21.15.78:0/3126224691 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd680089e70 0x7fd6800be760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.822+0000 7fd685f22700 1 --2- 172.21.15.78:0/3126224691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6800c2ed0 0x7fd6800b9930 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.822+0000 7fd685f22700 1 --2- 172.21.15.78:0/3126224691 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd66c06af00 0x7fd66c06d3c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.822+0000 7fd685f22700 1 --2- 172.21.15.78:0/3126224691 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd6800c4a50 0x7fd6800ba960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.822+0000 7fd685f22700 1 -- 172.21.15.78:0/3126224691 >> 172.21.15.78:0/3126224691 conn(0x7fd68006aab0 msgr2=0x7fd680070770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:21.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.822+0000 7fd685f22700 1 -- 172.21.15.78:0/3126224691 shutdown_connections 2023-10-14T11:58:21.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.822+0000 7f5a1bace700 1 -- 172.21.15.78:0/1784546362 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f5a14074b80 con 0x7f5a140c59e0 2023-10-14T11:58:21.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.823+0000 7fd685f22700 1 -- 172.21.15.78:0/3126224691 wait complete. 2023-10-14T11:58:21.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.823+0000 7f45c1aa4700 1 -- 172.21.15.78:0/2593509420 shutdown_connections 2023-10-14T11:58:21.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.823+0000 7f45c1aa4700 1 -- 172.21.15.78:0/2593509420 wait complete. 2023-10-14T11:58:21.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.823+0000 7f5a0a7fc700 1 -- 172.21.15.78:0/1784546362 <== mon.1 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5a0c057bf0 con 0x7f5a140c59e0 2023-10-14T11:58:21.831 INFO:teuthology.orchestra.run.smithi078.stdout:55834574852 2023-10-14T11:58:21.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.826+0000 7fd52e8e6700 1 -- 172.21.15.78:0/908389582 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fd5280aa000 con 0x7fd5280a7ba0 2023-10-14T11:58:21.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.827+0000 7f5a1bace700 1 -- 172.21.15.78:0/1784546362 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f5a0006b370 msgr2=0x7f5a0006d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.827+0000 7f5a1bace700 1 --2- 172.21.15.78:0/1784546362 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f5a0006b370 0x7f5a0006d830 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5a0400b010 tx=0x7f5a0400f000 comp rx=0 tx=0).stop 2023-10-14T11:58:21.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.827+0000 7f5a1bace700 1 -- 172.21.15.78:0/1784546362 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a140c59e0 msgr2=0x7f5a140aab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.827+0000 7f5a1bace700 1 --2- 172.21.15.78:0/1784546362 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a140c59e0 0x7f5a140aab00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c00c700 tx=0x7f5a0c00ca10 comp rx=0 tx=0).stop 2023-10-14T11:58:21.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.827+0000 7f5a1bace700 1 -- 172.21.15.78:0/1784546362 shutdown_connections 2023-10-14T11:58:21.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.827+0000 7f5a1bace700 1 --2- 172.21.15.78:0/1784546362 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f5a0006b370 0x7f5a0006d830 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.827+0000 7f5a1bace700 1 --2- 172.21.15.78:0/1784546362 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a140c59e0 0x7f5a140aab00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.827+0000 7f5a1bace700 1 --2- 172.21.15.78:0/1784546362 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5a140a7ba0 0x7f5a1414c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.827+0000 7f5a1bace700 1 --2- 172.21.15.78:0/1784546362 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a140c0830 0x7f5a141f1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.827+0000 7f5a1bace700 1 -- 172.21.15.78:0/1784546362 >> 172.21.15.78:0/1784546362 conn(0x7f5a1406aab0 msgr2=0x7f5a1414cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:21.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.827+0000 7fd524ff9700 1 -- 172.21.15.78:0/908389582 <== mon.2 v2:172.21.15.195:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd518057c90 con 0x7fd5280a7ba0 2023-10-14T11:58:21.834 INFO:teuthology.orchestra.run.smithi078.stdout:55834574852 2023-10-14T11:58:21.836 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.0 2023-10-14T11:58:21.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.830+0000 7efecd6c3700 1 -- 172.21.15.78:0/3599129615 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7efec80abe30 con 0x7efec80c2880 2023-10-14T11:58:21.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.831+0000 7f5a1bace700 1 -- 172.21.15.78:0/1784546362 shutdown_connections 2023-10-14T11:58:21.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.831+0000 7f5a1bace700 1 -- 172.21.15.78:0/1784546362 wait complete. 2023-10-14T11:58:21.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.831+0000 7efebffff700 1 -- 172.21.15.78:0/3599129615 <== mon.2 v2:172.21.15.195:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efeb0015090 con 0x7efec80c2880 2023-10-14T11:58:21.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.831+0000 7fd52e8e6700 1 -- 172.21.15.78:0/908389582 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd51406af00 msgr2=0x7fd51406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.831+0000 7fd52e8e6700 1 --2- 172.21.15.78:0/908389582 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd51406af00 0x7fd51406d3c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd51c000f60 tx=0x7fd51c007b10 comp rx=0 tx=0).stop 2023-10-14T11:58:21.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.831+0000 7fd52e8e6700 1 -- 172.21.15.78:0/908389582 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5280a7ba0 msgr2=0x7fd5280aa320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.831+0000 7fd52e8e6700 1 --2- 172.21.15.78:0/908389582 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5280a7ba0 0x7fd5280aa320 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd51800b4a0 tx=0x7fd51800b860 comp rx=0 tx=0).stop 2023-10-14T11:58:21.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.831+0000 7fd52e8e6700 1 -- 172.21.15.78:0/908389582 shutdown_connections 2023-10-14T11:58:21.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.831+0000 7fd52e8e6700 1 --2- 172.21.15.78:0/908389582 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5280c0830 0x7fd5280bd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.831+0000 7fd52e8e6700 1 --2- 172.21.15.78:0/908389582 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd51406af00 0x7fd51406d3c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.831+0000 7fd52e8e6700 1 --2- 172.21.15.78:0/908389582 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd5280706b0 0x7fd5280bf260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.831+0000 7fd52e8e6700 1 --2- 172.21.15.78:0/908389582 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5280a7ba0 0x7fd5280aa320 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.831+0000 7fd52e8e6700 1 -- 172.21.15.78:0/908389582 >> 172.21.15.78:0/908389582 conn(0x7fd52806aab0 msgr2=0x7fd5280c5eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:21.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.832+0000 7fd52e8e6700 1 -- 172.21.15.78:0/908389582 shutdown_connections 2023-10-14T11:58:21.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.832+0000 7fd52e8e6700 1 -- 172.21.15.78:0/908389582 wait complete. 2023-10-14T11:58:21.840 INFO:teuthology.orchestra.run.smithi078.stdout:55834574852 2023-10-14T11:58:21.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.834+0000 7efecd6c3700 1 -- 172.21.15.78:0/3599129615 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7efea806af00 msgr2=0x7efea806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.834+0000 7efecd6c3700 1 --2- 172.21.15.78:0/3599129615 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7efea806af00 0x7efea806d3c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7efeb8006f10 tx=0x7efeb8006e60 comp rx=0 tx=0).stop 2023-10-14T11:58:21.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.834+0000 7efecd6c3700 1 -- 172.21.15.78:0/3599129615 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efec80c2880 msgr2=0x7efec8153170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.834+0000 7efecd6c3700 1 --2- 172.21.15.78:0/3599129615 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efec80c2880 0x7efec8153170 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efeb00090b0 tx=0x7efeb0004750 comp rx=0 tx=0).stop 2023-10-14T11:58:21.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.835+0000 7efecd6c3700 1 -- 172.21.15.78:0/3599129615 shutdown_connections 2023-10-14T11:58:21.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.835+0000 7efecd6c3700 1 --2- 172.21.15.78:0/3599129615 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7efea806af00 0x7efea806d3c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.835+0000 7efecd6c3700 1 --2- 172.21.15.78:0/3599129615 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efec80bc780 0x7efec80b8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.835+0000 7efecd6c3700 1 --2- 172.21.15.78:0/3599129615 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efec80c2880 0x7efec8153170 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.835+0000 7efecd6c3700 1 --2- 172.21.15.78:0/3599129615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efec80c59e0 0x7efec80b8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.835+0000 7efecd6c3700 1 -- 172.21.15.78:0/3599129615 >> 172.21.15.78:0/3599129615 conn(0x7efec806aab0 msgr2=0x7efec8147940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:21.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.835+0000 7efecd6c3700 1 -- 172.21.15.78:0/3599129615 shutdown_connections 2023-10-14T11:58:21.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.835+0000 7efecd6c3700 1 -- 172.21.15.78:0/3599129615 wait complete. 2023-10-14T11:58:21.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.837+0000 7fe13e164700 1 -- 172.21.15.78:0/2384937077 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fe1380bf680 con 0x7fe13806b180 2023-10-14T11:58:21.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.837+0000 7fa9151c6700 1 -- 172.21.15.78:0/2928006316 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fa8f4005870 con 0x7fa9100710e0 2023-10-14T11:58:21.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.838+0000 7fe11ffff700 1 -- 172.21.15.78:0/2384937077 <== mon.1 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe1380bf680 con 0x7fe13806b180 2023-10-14T11:58:21.845 INFO:teuthology.orchestra.run.smithi078.stdout:55834574852 2023-10-14T11:58:21.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.838+0000 7fa9077fe700 1 -- 172.21.15.78:0/2928006316 <== mon.1 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa8f8053950 con 0x7fa9100710e0 2023-10-14T11:58:21.845 INFO:teuthology.orchestra.run.smithi078.stdout:55834574852 2023-10-14T11:58:21.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.842+0000 7fa9151c6700 1 -- 172.21.15.78:0/2928006316 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa8f006af00 msgr2=0x7fa8f006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.842+0000 7fa9151c6700 1 --2- 172.21.15.78:0/2928006316 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa8f006af00 0x7fa8f006d3c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa91014b660 tx=0x7fa900009210 comp rx=0 tx=0).stop 2023-10-14T11:58:21.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.842+0000 7fa9151c6700 1 -- 172.21.15.78:0/2928006316 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9100710e0 msgr2=0x7fa910146740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.842+0000 7fa9151c6700 1 --2- 172.21.15.78:0/2928006316 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9100710e0 0x7fa910146740 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa8f800c440 tx=0x7fa8f800c750 comp rx=0 tx=0).stop 2023-10-14T11:58:21.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.842+0000 7fa9151c6700 1 -- 172.21.15.78:0/2928006316 shutdown_connections 2023-10-14T11:58:21.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.842+0000 7fa9151c6700 1 --2- 172.21.15.78:0/2928006316 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9101523a0 0x7fa910141030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.842+0000 7fa9151c6700 1 --2- 172.21.15.78:0/2928006316 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa8f006af00 0x7fa8f006d3c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.842+0000 7fa9151c6700 1 --2- 172.21.15.78:0/2928006316 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9100710e0 0x7fa910146740 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.842+0000 7fa9151c6700 1 --2- 172.21.15.78:0/2928006316 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa91014ec80 0x7fa91014a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.842+0000 7fa9151c6700 1 -- 172.21.15.78:0/2928006316 >> 172.21.15.78:0/2928006316 conn(0x7fa9100c6310 msgr2=0x7fa9100c66f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:21.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.842+0000 7fa9151c6700 1 -- 172.21.15.78:0/2928006316 shutdown_connections 2023-10-14T11:58:21.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.842+0000 7fa9151c6700 1 -- 172.21.15.78:0/2928006316 wait complete. 2023-10-14T11:58:21.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.843+0000 7fe13e164700 1 -- 172.21.15.78:0/2384937077 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe12406af80 msgr2=0x7fe12406d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.843+0000 7fe13e164700 1 --2- 172.21.15.78:0/2384937077 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe12406af80 0x7fe12406d440 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe1380af2c0 tx=0x7fe12c009040 comp rx=0 tx=0).stop 2023-10-14T11:58:21.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.843+0000 7fe13e164700 1 -- 172.21.15.78:0/2384937077 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe13806b180 msgr2=0x7fe1380c13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:21.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.843+0000 7fe13e164700 1 --2- 172.21.15.78:0/2384937077 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe13806b180 0x7fe1380c13e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe1280090b0 tx=0x7fe128004750 comp rx=0 tx=0).stop 2023-10-14T11:58:21.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.845+0000 7fe13e164700 1 -- 172.21.15.78:0/2384937077 shutdown_connections 2023-10-14T11:58:21.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.845+0000 7fe13e164700 1 --2- 172.21.15.78:0/2384937077 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1380c4f40 0x7fe1380c09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.845+0000 7fe13e164700 1 --2- 172.21.15.78:0/2384937077 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe12406af80 0x7fe12406d440 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.845+0000 7fe13e164700 1 --2- 172.21.15.78:0/2384937077 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe13806b180 0x7fe1380c13e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.845+0000 7fe13e164700 1 --2- 172.21.15.78:0/2384937077 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe1380afad0 0x7fe138150e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:21.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.845+0000 7fe13e164700 1 -- 172.21.15.78:0/2384937077 >> 172.21.15.78:0/2384937077 conn(0x7fe1380c6e20 msgr2=0x7fe138154ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:21.852 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.2 2023-10-14T11:58:21.853 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.3 2023-10-14T11:58:21.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.846+0000 7fe13e164700 1 -- 172.21.15.78:0/2384937077 shutdown_connections 2023-10-14T11:58:21.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:21.846+0000 7fe13e164700 1 -- 172.21.15.78:0/2384937077 wait complete. 2023-10-14T11:58:21.858 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.1 2023-10-14T11:58:21.860 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.7 2023-10-14T11:58:21.863 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.6 2023-10-14T11:58:21.863 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.5 2023-10-14T11:58:21.866 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.4 2023-10-14T11:58:22.837 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-14T11:58:22.853 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-14T11:58:22.854 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-14T11:58:22.859 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-14T11:58:22.861 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-14T11:58:22.864 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-14T11:58:22.864 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-14T11:58:22.867 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-14T11:58:23.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.030+0000 7fcc7c665700 1 Processor -- start 2023-10-14T11:58:23.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.030+0000 7fcc7c665700 1 -- start start 2023-10-14T11:58:23.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.032+0000 7fcc7c665700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcc740c0830 0x7fcc74072a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.032+0000 7fcc7c665700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcc740c2880 0x7fcc740a7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.032+0000 7fcc7c665700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc740c18b0 con 0x7fcc740706b0 2023-10-14T11:58:23.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.032+0000 7fcc7c665700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc740c15d0 con 0x7fcc740c2880 2023-10-14T11:58:23.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.032+0000 7fcc7c665700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc740c3f30 con 0x7fcc740c0830 2023-10-14T11:58:23.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.033+0000 7fcc7a401700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcc740c0830 0x7fcc74072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.033+0000 7fcc7a401700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcc740c0830 0x7fcc74072a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.78:44474/0 (socket says 172.21.15.78:44474) 2023-10-14T11:58:23.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.033+0000 7fcc7a401700 1 -- 172.21.15.78:0/4176819383 learned_addr learned my addr 172.21.15.78:0/4176819383 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:23.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.033+0000 7fcc79c00700 1 --2- 172.21.15.78:0/4176819383 >> v2:172.21.15.78:3300/0 conn(0x7fcc740c2880 0x7fcc740a7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.034+0000 7fcc79c00700 1 -- 172.21.15.78:0/4176819383 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcc740c0830 msgr2=0x7fcc74072a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:23.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.034+0000 7fcc79c00700 1 --2- 172.21.15.78:0/4176819383 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcc740c0830 0x7fcc74072a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.034+0000 7fcc79c00700 1 -- 172.21.15.78:0/4176819383 >> v1:172.21.15.78:6789/0 conn(0x7fcc740706b0 legacy=0x7fcc740c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:23.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.034+0000 7fcc79c00700 1 -- 172.21.15.78:0/4176819383 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc740c4210 con 0x7fcc740c2880 2023-10-14T11:58:23.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.034+0000 7fcc79c00700 1 --2- 172.21.15.78:0/4176819383 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc740c2880 0x7fcc740a7ba0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fcc70009840 tx=0x7fcc70009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f714578154ac5f83 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:23.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.035+0000 7fcc793ff700 1 -- 172.21.15.78:0/4176819383 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc70004230 con 0x7fcc740c2880 2023-10-14T11:58:23.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.035+0000 7fcc793ff700 1 -- 172.21.15.78:0/4176819383 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc70004390 con 0x7fcc740c2880 2023-10-14T11:58:23.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.035+0000 7fcc793ff700 1 -- 172.21.15.78:0/4176819383 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc700036a0 con 0x7fcc740c2880 2023-10-14T11:58:23.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.036+0000 7fcc7c665700 1 -- 172.21.15.78:0/4176819383 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc740c2880 msgr2=0x7fcc740a7ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.036+0000 7fcc7c665700 1 --2- 172.21.15.78:0/4176819383 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc740c2880 0x7fcc740a7ba0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fcc70009840 tx=0x7fcc70009b50 comp rx=0 tx=0).stop 2023-10-14T11:58:23.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.037+0000 7fcc7c665700 1 -- 172.21.15.78:0/4176819383 shutdown_connections 2023-10-14T11:58:23.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.037+0000 7fcc7c665700 1 --2- 172.21.15.78:0/4176819383 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcc740c0830 0x7fcc74072a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.037+0000 7fcc7c665700 1 --2- 172.21.15.78:0/4176819383 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc740c2880 0x7fcc740a7ba0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.037+0000 7fcc7c665700 1 -- 172.21.15.78:0/4176819383 >> 172.21.15.78:0/4176819383 conn(0x7fcc7406aab0 msgr2=0x7fcc7406aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:23.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.037+0000 7fcc7c665700 1 -- 172.21.15.78:0/4176819383 shutdown_connections 2023-10-14T11:58:23.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.037+0000 7fcc7c665700 1 -- 172.21.15.78:0/4176819383 wait complete. 2023-10-14T11:58:23.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.039+0000 7fcc7c665700 1 Processor -- start 2023-10-14T11:58:23.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.039+0000 7fcc7c665700 1 -- start start 2023-10-14T11:58:23.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.040+0000 7fcc7c665700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcc740706b0 0x7fcc740be600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.040+0000 7fcc7c665700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcc740c0830 0x7fcc740bd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.040+0000 7fcc7c665700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc740c2880 0x7fcc740bd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.040+0000 7fcc7c665700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc740c00f0 con 0x7fcc740c2880 2023-10-14T11:58:23.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.040+0000 7fcc7c665700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc740ab820 con 0x7fcc740706b0 2023-10-14T11:58:23.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.040+0000 7fcc7c665700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc740abb00 con 0x7fcc740c0830 2023-10-14T11:58:23.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.041+0000 7fcc7ac02700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc740c2880 0x7fcc740bd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.041+0000 7fcc7ac02700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc740c2880 0x7fcc740bd8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44814/0 (socket says 172.21.15.78:44814) 2023-10-14T11:58:23.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.041+0000 7fcc7ac02700 1 -- 172.21.15.78:0/757927800 learned_addr learned my addr 172.21.15.78:0/757927800 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:23.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.041+0000 7fcc7a401700 1 --2- 172.21.15.78:0/757927800 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcc740706b0 0x7fcc740be600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.041+0000 7fcc79c00700 1 --2- 172.21.15.78:0/757927800 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcc740c0830 0x7fcc740bd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.041+0000 7fcc7ac02700 1 -- 172.21.15.78:0/757927800 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcc740c0830 msgr2=0x7fcc740bd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.042+0000 7fcc7ac02700 1 --2- 172.21.15.78:0/757927800 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcc740c0830 0x7fcc740bd3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.042+0000 7fcc7ac02700 1 -- 172.21.15.78:0/757927800 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcc740706b0 msgr2=0x7fcc740be600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.042+0000 7fcc7ac02700 1 --2- 172.21.15.78:0/757927800 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcc740706b0 0x7fcc740be600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.042+0000 7fcc7ac02700 1 -- 172.21.15.78:0/757927800 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc700094a0 con 0x7fcc740c2880 2023-10-14T11:58:23.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.042+0000 7fcc7ac02700 1 --2- 172.21.15.78:0/757927800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc740c2880 0x7fcc740bd8e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fcc6400c450 tx=0x7fcc6400c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:23.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.042+0000 7fcc7a401700 1 --2- 172.21.15.78:0/757927800 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcc740706b0 0x7fcc740be600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:23.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.043+0000 7fcc6affd700 1 -- 172.21.15.78:0/757927800 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc64011790 con 0x7fcc740c2880 2023-10-14T11:58:23.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.043+0000 7fcc6affd700 1 -- 172.21.15.78:0/757927800 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc64011930 con 0x7fcc740c2880 2023-10-14T11:58:23.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.043+0000 7fcc6affd700 1 -- 172.21.15.78:0/757927800 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc64011ce0 con 0x7fcc740c2880 2023-10-14T11:58:23.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.043+0000 7fcc7c665700 1 -- 172.21.15.78:0/757927800 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc740a69d0 con 0x7fcc740c2880 2023-10-14T11:58:23.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.043+0000 7fcc7c665700 1 -- 172.21.15.78:0/757927800 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc7407daf0 con 0x7fcc740c2880 2023-10-14T11:58:23.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.047+0000 7fcc6affd700 1 -- 172.21.15.78:0/757927800 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fcc6401c460 con 0x7fcc740c2880 2023-10-14T11:58:23.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.048+0000 7fcc6affd700 1 --2- 172.21.15.78:0/757927800 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fcc5806afa0 0x7fcc5806d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.048+0000 7fcc6affd700 1 -- 172.21.15.78:0/757927800 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcc64089300 con 0x7fcc740c2880 2023-10-14T11:58:23.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.050+0000 7fcc7a401700 1 --2- 172.21.15.78:0/757927800 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fcc5806afa0 0x7fcc5806d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.051+0000 7fcc7a401700 1 --2- 172.21.15.78:0/757927800 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fcc5806afa0 0x7fcc5806d460 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c009c60 tx=0x7fcc6c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:23.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.051+0000 7fcc7c665700 1 -- 172.21.15.78:0/757927800 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc5c0052f0 con 0x7fcc740c2880 2023-10-14T11:58:23.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.061+0000 7fcc6affd700 1 -- 172.21.15.78:0/757927800 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcc64054390 con 0x7fcc740c2880 2023-10-14T11:58:23.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.098+0000 7f49d62c9700 1 Processor -- start 2023-10-14T11:58:23.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.098+0000 7f49d62c9700 1 -- start start 2023-10-14T11:58:23.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.097+0000 7f6acc3b5700 1 Processor -- start 2023-10-14T11:58:23.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.098+0000 7f6acc3b5700 1 -- start start 2023-10-14T11:58:23.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.099+0000 7f6acc3b5700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6ac406df80 0x7f6ac40c46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.099+0000 7f6acc3b5700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6ac40a4ea0 0x7f6ac41657c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.099+0000 7f6acc3b5700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ac40b9760 con 0x7f6ac40c2ed0 2023-10-14T11:58:23.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.099+0000 7f6acc3b5700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ac408a9a0 con 0x7f6ac40a4ea0 2023-10-14T11:58:23.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.099+0000 7f6acc3b5700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ac40c0830 con 0x7f6ac406df80 2023-10-14T11:58:23.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.099+0000 7f6ac9950700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6ac406df80 0x7f6ac40c46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.101+0000 7f9284324700 1 Processor -- start 2023-10-14T11:58:23.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.100+0000 7f49d62c9700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f49d00c3fd0 0x7f49d00c4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.100+0000 7f49d62c9700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f49d00c4970 0x7f49d01657c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.100+0000 7f49d62c9700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49d00b9760 con 0x7f49d00a4ec0 2023-10-14T11:58:23.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.100+0000 7f49d62c9700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49d008a9a0 con 0x7f49d00c4970 2023-10-14T11:58:23.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.100+0000 7f49d62c9700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49d00c0830 con 0x7f49d00c3fd0 2023-10-14T11:58:23.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.100+0000 7f49cffff700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f49d00a4ec0 0x7f49d00c2c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49700/0 (socket says 172.21.15.78:49700) 2023-10-14T11:58:23.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.100+0000 7f49cffff700 1 -- 172.21.15.78:0/4226561891 learned_addr learned my addr 172.21.15.78:0/4226561891 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:23.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.100+0000 7f49cf7fe700 1 --2- 172.21.15.78:0/4226561891 >> v2:172.21.15.195:3300/0 conn(0x7f49d00c3fd0 0x7f49d00c4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.100+0000 7f49cf7fe700 1 -- 172.21.15.78:0/4226561891 >> v2:172.21.15.78:3300/0 conn(0x7f49d00c4970 msgr2=0x7f49d01657c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:23.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.100+0000 7f49cf7fe700 1 --2- 172.21.15.78:0/4226561891 >> v2:172.21.15.78:3300/0 conn(0x7f49d00c4970 0x7f49d01657c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.100+0000 7f49cf7fe700 1 -- 172.21.15.78:0/4226561891 >> v1:172.21.15.78:6789/0 conn(0x7f49d00a4ec0 legacy=0x7f49d00c2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.100+0000 7f49cf7fe700 1 -- 172.21.15.78:0/4226561891 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d00c0a80 con 0x7f49d00c3fd0 2023-10-14T11:58:23.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.101+0000 7f49cf7fe700 1 --2- 172.21.15.78:0/4226561891 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f49d00c3fd0 0x7f49d00c4430 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f49d00c0400 tx=0x7f49c0009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbe5b6c75cf2c0b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:23.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.101+0000 7f49ceffd700 1 -- 172.21.15.78:0/4226561891 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c000f940 con 0x7f49d00c3fd0 2023-10-14T11:58:23.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.099+0000 7fb1ad488700 1 Processor -- start 2023-10-14T11:58:23.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.100+0000 7fb1ad488700 1 -- start start 2023-10-14T11:58:23.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.101+0000 7fb1ad488700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb1a806df80 0x7fb1a80c46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.101+0000 7fb1ad488700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb1a80a4ea0 0x7fb1a81657c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.101+0000 7fb1ad488700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a80b9760 con 0x7fb1a80c2ed0 2023-10-14T11:58:23.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.101+0000 7fb1ad488700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1a808a9a0 con 0x7fb1a806df80 2023-10-14T11:58:23.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.101+0000 7fb1ad488700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1a80c0830 con 0x7fb1a80a4ea0 2023-10-14T11:58:23.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.101+0000 7fb1a77fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb1a80a4ea0 0x7fb1a81657c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.101+0000 7fb1a67fc700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb1a806df80 0x7fb1a80c46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.100+0000 7f6aca151700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f6ac40c2ed0 0x7f6ac40c3fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49694/0 (socket says 172.21.15.78:49694) 2023-10-14T11:58:23.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.100+0000 7f6aca151700 1 -- 172.21.15.78:0/1899745999 learned_addr learned my addr 172.21.15.78:0/1899745999 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:23.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.100+0000 7f6ac9950700 1 -- 172.21.15.78:0/1899745999 >> v2:172.21.15.78:3300/0 conn(0x7f6ac40a4ea0 msgr2=0x7f6ac41657c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:23.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.100+0000 7f6ac9950700 1 --2- 172.21.15.78:0/1899745999 >> v2:172.21.15.78:3300/0 conn(0x7f6ac40a4ea0 0x7f6ac41657c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.100+0000 7f6ac9950700 1 -- 172.21.15.78:0/1899745999 >> v1:172.21.15.78:6789/0 conn(0x7f6ac40c2ed0 legacy=0x7f6ac40c3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.100+0000 7f6ac9950700 1 -- 172.21.15.78:0/1899745999 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac408abf0 con 0x7f6ac406df80 2023-10-14T11:58:23.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.101+0000 7f6ac9950700 1 --2- 172.21.15.78:0/1899745999 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ac406df80 0x7f6ac40c46e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6ac40c25e0 tx=0x7f6ab400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4da153947e687ca7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:23.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.101+0000 7fb1a67fc700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb1a806df80 0x7fb1a80c46e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44834/0 (socket says 172.21.15.78:44834) 2023-10-14T11:58:23.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.101+0000 7fb1a67fc700 1 -- 172.21.15.78:0/627115745 learned_addr learned my addr 172.21.15.78:0/627115745 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:23.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.101+0000 7f49ceffd700 1 -- 172.21.15.78:0/4226561891 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c000faa0 con 0x7f49d00c3fd0 2023-10-14T11:58:23.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.101+0000 7f49ceffd700 1 -- 172.21.15.78:0/4226561891 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c000fe30 con 0x7f49d00c3fd0 2023-10-14T11:58:23.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.101+0000 7f6ac914f700 1 -- 172.21.15.78:0/1899745999 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab400e070 con 0x7f6ac406df80 2023-10-14T11:58:23.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.101+0000 7f6ac914f700 1 -- 172.21.15.78:0/1899745999 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab400bbd0 con 0x7f6ac406df80 2023-10-14T11:58:23.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.101+0000 7f6ac914f700 1 -- 172.21.15.78:0/1899745999 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab4012430 con 0x7f6ac406df80 2023-10-14T11:58:23.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.101+0000 7f9284324700 1 -- start start 2023-10-14T11:58:23.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.102+0000 7fb1a67fc700 1 -- 172.21.15.78:0/627115745 >> v2:172.21.15.195:3300/0 conn(0x7fb1a80a4ea0 msgr2=0x7fb1a81657c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:23.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.102+0000 7fb1a67fc700 1 --2- 172.21.15.78:0/627115745 >> v2:172.21.15.195:3300/0 conn(0x7fb1a80a4ea0 0x7fb1a81657c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.102+0000 7fb1a67fc700 1 -- 172.21.15.78:0/627115745 >> v1:172.21.15.195:6789/0 conn(0x7fb1a80c2ed0 legacy=0x7fb1a80c3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:23.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.102+0000 7fb1a67fc700 1 -- 172.21.15.78:0/627115745 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1a808abf0 con 0x7fb1a806df80 2023-10-14T11:58:23.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.102+0000 7f49d62c9700 1 -- 172.21.15.78:0/4226561891 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f49d00c3fd0 msgr2=0x7f49d00c4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.102+0000 7f49d62c9700 1 --2- 172.21.15.78:0/4226561891 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f49d00c3fd0 0x7f49d00c4430 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f49d00c0400 tx=0x7f49c0009b20 comp rx=0 tx=0).stop 2023-10-14T11:58:23.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.102+0000 7f49d62c9700 1 -- 172.21.15.78:0/4226561891 shutdown_connections 2023-10-14T11:58:23.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.102+0000 7f49d62c9700 1 --2- 172.21.15.78:0/4226561891 >> v2:172.21.15.78:3300/0 conn(0x7f49d00c4970 0x7f49d01657c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.102+0000 7f49d62c9700 1 --2- 172.21.15.78:0/4226561891 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f49d00c3fd0 0x7f49d00c4430 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.102+0000 7f49d62c9700 1 -- 172.21.15.78:0/4226561891 >> 172.21.15.78:0/4226561891 conn(0x7f49d006aab0 msgr2=0x7f49d006aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:23.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.102+0000 7f6acc3b5700 1 -- 172.21.15.78:0/1899745999 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ac406df80 msgr2=0x7f6ac40c46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.102+0000 7f6acc3b5700 1 --2- 172.21.15.78:0/1899745999 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ac406df80 0x7f6ac40c46e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6ac40c25e0 tx=0x7f6ab400b330 comp rx=0 tx=0).stop 2023-10-14T11:58:23.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.102+0000 7f9284324700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f927c0c65c0 0x7f927c0a7930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.102+0000 7f9284324700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f927c072cf0 0x7f927c0c4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.102+0000 7f9284324700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f927c0c3fe0 con 0x7f927c072910 2023-10-14T11:58:23.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.102+0000 7f9284324700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f927c0c0bf0 con 0x7f927c072cf0 2023-10-14T11:58:23.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.102+0000 7f9284324700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f927c0c0ed0 con 0x7f927c0c65c0 2023-10-14T11:58:23.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.102+0000 7f6acc3b5700 1 -- 172.21.15.78:0/1899745999 shutdown_connections 2023-10-14T11:58:23.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.102+0000 7f6acc3b5700 1 --2- 172.21.15.78:0/1899745999 >> v2:172.21.15.78:3300/0 conn(0x7f6ac40a4ea0 0x7f6ac41657c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.102+0000 7f6acc3b5700 1 --2- 172.21.15.78:0/1899745999 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ac406df80 0x7f6ac40c46e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.102+0000 7f6acc3b5700 1 -- 172.21.15.78:0/1899745999 >> 172.21.15.78:0/1899745999 conn(0x7f6ac406aab0 msgr2=0x7f6ac406aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:23.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.103+0000 7f92820c0700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f927c072910 0x7f927c0c5570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.78:39518/0 (socket says 172.21.15.78:39518) 2023-10-14T11:58:23.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.103+0000 7f92820c0700 1 -- 172.21.15.78:0/2784212486 learned_addr learned my addr 172.21.15.78:0/2784212486 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:23.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.103+0000 7f49d62c9700 1 -- 172.21.15.78:0/4226561891 shutdown_connections 2023-10-14T11:58:23.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.103+0000 7f49d62c9700 1 -- 172.21.15.78:0/4226561891 wait complete. 2023-10-14T11:58:23.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.103+0000 7f92818bf700 1 --2- 172.21.15.78:0/2784212486 >> v2:172.21.15.195:3300/0 conn(0x7f927c0c65c0 0x7f927c0a7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.103+0000 7f92818bf700 1 -- 172.21.15.78:0/2784212486 >> v2:172.21.15.78:3300/0 conn(0x7f927c072cf0 msgr2=0x7f927c0c4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:23.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.103+0000 7f92818bf700 1 --2- 172.21.15.78:0/2784212486 >> v2:172.21.15.78:3300/0 conn(0x7f927c072cf0 0x7f927c0c4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.103+0000 7f92818bf700 1 -- 172.21.15.78:0/2784212486 >> v1:172.21.15.195:6789/0 conn(0x7f927c072910 legacy=0x7f927c0c5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.103+0000 7f92818bf700 1 -- 172.21.15.78:0/2784212486 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f927c0c11b0 con 0x7f927c0c65c0 2023-10-14T11:58:23.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.103+0000 7f6acc3b5700 1 -- 172.21.15.78:0/1899745999 shutdown_connections 2023-10-14T11:58:23.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.103+0000 7f6acc3b5700 1 -- 172.21.15.78:0/1899745999 wait complete. 2023-10-14T11:58:23.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.104+0000 7f92818bf700 1 --2- 172.21.15.78:0/2784212486 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f927c0c65c0 0x7f927c0a7930 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f927c0c26f0 tx=0x7f9278009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8361073c20c21da6 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:23.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.103+0000 7fb1a67fc700 1 --2- 172.21.15.78:0/627115745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1a806df80 0x7fb1a80c46e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb1a806c1c0 tx=0x7fb19c00d330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c90a53d665ff3f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:23.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.103+0000 7fb1a5ffb700 1 -- 172.21.15.78:0/627115745 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19c010070 con 0x7fb1a806df80 2023-10-14T11:58:23.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.103+0000 7fb1a5ffb700 1 -- 172.21.15.78:0/627115745 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb19c00da40 con 0x7fb1a806df80 2023-10-14T11:58:23.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.103+0000 7fb1a5ffb700 1 -- 172.21.15.78:0/627115745 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19c00ddb0 con 0x7fb1a806df80 2023-10-14T11:58:23.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.104+0000 7f92810be700 1 -- 172.21.15.78:0/2784212486 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9278004230 con 0x7f927c0c65c0 2023-10-14T11:58:23.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.104+0000 7f92810be700 1 -- 172.21.15.78:0/2784212486 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9278004390 con 0x7f927c0c65c0 2023-10-14T11:58:23.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.104+0000 7f92810be700 1 -- 172.21.15.78:0/2784212486 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92780036a0 con 0x7f927c0c65c0 2023-10-14T11:58:23.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.104+0000 7f49d62c9700 1 Processor -- start 2023-10-14T11:58:23.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.105+0000 7f49d62c9700 1 -- start start 2023-10-14T11:58:23.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.104+0000 7fb1ad488700 1 -- 172.21.15.78:0/627115745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1a806df80 msgr2=0x7fb1a80c46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.104+0000 7fb1ad488700 1 --2- 172.21.15.78:0/627115745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1a806df80 0x7fb1a80c46e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb1a806c1c0 tx=0x7fb19c00d330 comp rx=0 tx=0).stop 2023-10-14T11:58:23.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.105+0000 7f6acc3b5700 1 Processor -- start 2023-10-14T11:58:23.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.105+0000 7fb1ad488700 1 -- 172.21.15.78:0/627115745 shutdown_connections 2023-10-14T11:58:23.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.105+0000 7fb1ad488700 1 --2- 172.21.15.78:0/627115745 >> v2:172.21.15.195:3300/0 conn(0x7fb1a80a4ea0 0x7fb1a81657c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.105+0000 7fb1ad488700 1 --2- 172.21.15.78:0/627115745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1a806df80 0x7fb1a80c46e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.105+0000 7fb1ad488700 1 -- 172.21.15.78:0/627115745 >> 172.21.15.78:0/627115745 conn(0x7fb1a806aab0 msgr2=0x7fb1a806aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:23.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.105+0000 7f9284324700 1 -- 172.21.15.78:0/2784212486 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f927c0c65c0 msgr2=0x7f927c0a7930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.105+0000 7f9284324700 1 --2- 172.21.15.78:0/2784212486 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f927c0c65c0 0x7f927c0a7930 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f927c0c26f0 tx=0x7f9278009b20 comp rx=0 tx=0).stop 2023-10-14T11:58:23.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.105+0000 7f9284324700 1 -- 172.21.15.78:0/2784212486 shutdown_connections 2023-10-14T11:58:23.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.105+0000 7f9284324700 1 --2- 172.21.15.78:0/2784212486 >> v2:172.21.15.78:3300/0 conn(0x7f927c072cf0 0x7f927c0c4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.105+0000 7f9284324700 1 --2- 172.21.15.78:0/2784212486 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f927c0c65c0 0x7f927c0a7930 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.105+0000 7f9284324700 1 -- 172.21.15.78:0/2784212486 >> 172.21.15.78:0/2784212486 conn(0x7f927c06af90 msgr2=0x7f927c0c5eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:23.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.105+0000 7f9284324700 1 -- 172.21.15.78:0/2784212486 shutdown_connections 2023-10-14T11:58:23.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.105+0000 7f9284324700 1 -- 172.21.15.78:0/2784212486 wait complete. 2023-10-14T11:58:23.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.105+0000 7f49d62c9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49d00a4ec0 0x7f49d00beca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.105+0000 7f49d62c9700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f49d00c3fd0 0x7f49d00893a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.105+0000 7f49d62c9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f49d00c4970 0x7f49d0088810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.105+0000 7f49d62c9700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d00bf7f0 con 0x7f49d00a4ec0 2023-10-14T11:58:23.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.106+0000 7f49d62c9700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d0200760 con 0x7f49d00c4970 2023-10-14T11:58:23.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.106+0000 7f49d62c9700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49d00ab650 con 0x7f49d00c3fd0 2023-10-14T11:58:23.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.106+0000 7f49cffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49d00a4ec0 0x7f49d00beca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.106+0000 7f49cffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49d00a4ec0 0x7f49d00beca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44844/0 (socket says 172.21.15.78:44844) 2023-10-14T11:58:23.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.106+0000 7f49cffff700 1 -- 172.21.15.78:0/2393682697 learned_addr learned my addr 172.21.15.78:0/2393682697 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:23.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.106+0000 7f49cf7fe700 1 --2- 172.21.15.78:0/2393682697 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f49d00c3fd0 0x7f49d00893a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.106+0000 7f49cf7fe700 1 -- 172.21.15.78:0/2393682697 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f49d00c4970 msgr2=0x7f49d0088810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.106+0000 7f49cf7fe700 1 --2- 172.21.15.78:0/2393682697 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f49d00c4970 0x7f49d0088810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.106+0000 7f49cf7fe700 1 -- 172.21.15.78:0/2393682697 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49d00a4ec0 msgr2=0x7f49d00beca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.106+0000 7f49cf7fe700 1 --2- 172.21.15.78:0/2393682697 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49d00a4ec0 0x7f49d00beca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.106+0000 7f49cf7fe700 1 -- 172.21.15.78:0/2393682697 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c00094a0 con 0x7f49d00c3fd0 2023-10-14T11:58:23.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.105+0000 7f6acc3b5700 1 -- start start 2023-10-14T11:58:23.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.106+0000 7f6acc3b5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ac406df80 0x7f6ac4200000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.106+0000 7f6acc3b5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ac40a4ea0 0x7f6ac40830f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.106+0000 7f6acc3b5700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ac40c2ed0 0x7f6ac4085410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.106+0000 7f6acc3b5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac4200540 con 0x7f6ac406df80 2023-10-14T11:58:23.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.106+0000 7f6acc3b5700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac4082c60 con 0x7f6ac40a4ea0 2023-10-14T11:58:23.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.106+0000 7f6acc3b5700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac40be430 con 0x7f6ac40c2ed0 2023-10-14T11:58:23.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.106+0000 7f6aca151700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ac406df80 0x7f6ac4200000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.106+0000 7f6aca151700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ac406df80 0x7f6ac4200000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44848/0 (socket says 172.21.15.78:44848) 2023-10-14T11:58:23.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.106+0000 7f6aca151700 1 -- 172.21.15.78:0/181775399 learned_addr learned my addr 172.21.15.78:0/181775399 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:23.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.106+0000 7f6ac9950700 1 --2- 172.21.15.78:0/181775399 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ac40a4ea0 0x7f6ac40830f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.106+0000 7f6aca952700 1 --2- 172.21.15.78:0/181775399 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ac40c2ed0 0x7f6ac4085410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.107+0000 7f49cf7fe700 1 --2- 172.21.15.78:0/2393682697 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f49d00c3fd0 0x7f49d00893a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f49c0018000 tx=0x7f49c0004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:23.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.107+0000 7fb1ad488700 1 -- 172.21.15.78:0/627115745 shutdown_connections 2023-10-14T11:58:23.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.107+0000 7fb1ad488700 1 -- 172.21.15.78:0/627115745 wait complete. 2023-10-14T11:58:23.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.107+0000 7f6aca151700 1 -- 172.21.15.78:0/181775399 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ac40c2ed0 msgr2=0x7f6ac4085410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.107+0000 7f6aca151700 1 --2- 172.21.15.78:0/181775399 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ac40c2ed0 0x7f6ac4085410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.107+0000 7f6aca151700 1 -- 172.21.15.78:0/181775399 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ac40a4ea0 msgr2=0x7f6ac40830f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.107+0000 7f6aca151700 1 --2- 172.21.15.78:0/181775399 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ac40a4ea0 0x7f6ac40830f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.107+0000 7f6aca151700 1 -- 172.21.15.78:0/181775399 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab4009a70 con 0x7f6ac406df80 2023-10-14T11:58:23.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.107+0000 7f6aca151700 1 --2- 172.21.15.78:0/181775399 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ac406df80 0x7f6ac4200000 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6ac000c4c0 tx=0x7f6ac000c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:23.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.107+0000 7f9284324700 1 Processor -- start 2023-10-14T11:58:23.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.107+0000 7f9284324700 1 -- start start 2023-10-14T11:58:23.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.108+0000 7f9284324700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f927c072910 0x7f927c1e0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.108+0000 7f9284324700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f927c0c65c0 0x7f927c1f1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.108+0000 7f9284324700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f927c1ded30 0x7f927c1f1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.108+0000 7f9284324700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f927c0a9440 con 0x7f927c0c65c0 2023-10-14T11:58:23.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.108+0000 7f9284324700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f927c0a7540 con 0x7f927c072910 2023-10-14T11:58:23.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.108+0000 7f9284324700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f927c0b9a10 con 0x7f927c1ded30 2023-10-14T11:58:23.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.108+0000 7f92828c1700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f927c1ded30 0x7f927c1f1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.108+0000 7f92828c1700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f927c1ded30 0x7f927c1f1b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.78:44524/0 (socket says 172.21.15.78:44524) 2023-10-14T11:58:23.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.108+0000 7f92828c1700 1 -- 172.21.15.78:0/3595420768 learned_addr learned my addr 172.21.15.78:0/3595420768 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:23.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.108+0000 7f92828c1700 1 -- 172.21.15.78:0/3595420768 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f927c072910 msgr2=0x7f927c1e0bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T11:58:23.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.108+0000 7f92828c1700 1 --2- 172.21.15.78:0/3595420768 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f927c072910 0x7f927c1e0bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.108+0000 7f92828c1700 1 -- 172.21.15.78:0/3595420768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f927c0c65c0 msgr2=0x7f927c1f1650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T11:58:23.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.108+0000 7f92828c1700 1 --2- 172.21.15.78:0/3595420768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f927c0c65c0 0x7f927c1f1650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.108+0000 7f92828c1700 1 -- 172.21.15.78:0/3595420768 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92780094a0 con 0x7f927c1ded30 2023-10-14T11:58:23.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.108+0000 7f92828c1700 1 --2- 172.21.15.78:0/3595420768 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f927c1ded30 0x7f927c1f1b90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f927400ba10 tx=0x7f927400bdd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:23.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.109+0000 7fb1ad488700 1 Processor -- start 2023-10-14T11:58:23.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.109+0000 7f9272ffd700 1 -- 172.21.15.78:0/3595420768 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f927400c9b0 con 0x7f927c1ded30 2023-10-14T11:58:23.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.109+0000 7f9272ffd700 1 -- 172.21.15.78:0/3595420768 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f927400cb10 con 0x7f927c1ded30 2023-10-14T11:58:23.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.109+0000 7f9272ffd700 1 -- 172.21.15.78:0/3595420768 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9274014440 con 0x7f927c1ded30 2023-10-14T11:58:23.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.109+0000 7fb1ad488700 1 -- start start 2023-10-14T11:58:23.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.109+0000 7f9284324700 1 -- 172.21.15.78:0/3595420768 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f927c0a8fb0 con 0x7f927c1ded30 2023-10-14T11:58:23.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.109+0000 7f9284324700 1 -- 172.21.15.78:0/3595420768 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f927c0b9f40 con 0x7f927c1ded30 2023-10-14T11:58:23.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.110+0000 7fb1ad488700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb1a806df80 0x7fb1a82003f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.110+0000 7fb1ad488700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1a80a4ea0 0x7fb1a8200930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.110+0000 7fb1ad488700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb1a80c2ed0 0x7fb1a81fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.110+0000 7fb1ad488700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a80bf810 con 0x7fb1a80a4ea0 2023-10-14T11:58:23.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.110+0000 7fb1ad488700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a81ffe50 con 0x7fb1a806df80 2023-10-14T11:58:23.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.110+0000 7fb1ad488700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a80a9580 con 0x7fb1a80c2ed0 2023-10-14T11:58:23.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.110+0000 7fb1a67fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1a80a4ea0 0x7fb1a8200930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.110+0000 7fb1a67fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1a80a4ea0 0x7fb1a8200930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44858/0 (socket says 172.21.15.78:44858) 2023-10-14T11:58:23.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.110+0000 7fb1a67fc700 1 -- 172.21.15.78:0/1667041500 learned_addr learned my addr 172.21.15.78:0/1667041500 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:23.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.110+0000 7fb1a77fe700 1 --2- 172.21.15.78:0/1667041500 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb1a80c2ed0 0x7fb1a81fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.110+0000 7fb1a6ffd700 1 --2- 172.21.15.78:0/1667041500 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb1a806df80 0x7fb1a82003f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.110+0000 7fb1a67fc700 1 -- 172.21.15.78:0/1667041500 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb1a80c2ed0 msgr2=0x7fb1a81fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.110+0000 7fb1a67fc700 1 --2- 172.21.15.78:0/1667041500 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb1a80c2ed0 0x7fb1a81fc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.110+0000 7fb1a67fc700 1 -- 172.21.15.78:0/1667041500 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb1a806df80 msgr2=0x7fb1a82003f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.110+0000 7fb1a67fc700 1 --2- 172.21.15.78:0/1667041500 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb1a806df80 0x7fb1a82003f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.110+0000 7fb1a67fc700 1 -- 172.21.15.78:0/1667041500 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb19c0085d0 con 0x7fb1a80a4ea0 2023-10-14T11:58:23.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.111+0000 7fb1a67fc700 1 --2- 172.21.15.78:0/1667041500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1a80a4ea0 0x7fb1a8200930 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb19c00d370 tx=0x7fb19c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:23.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.111+0000 7fb18f7fe700 1 -- 172.21.15.78:0/1667041500 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19c004130 con 0x7fb1a80a4ea0 2023-10-14T11:58:23.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.111+0000 7fb18f7fe700 1 -- 172.21.15.78:0/1667041500 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb19c004290 con 0x7fb1a80a4ea0 2023-10-14T11:58:23.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.111+0000 7fb18f7fe700 1 -- 172.21.15.78:0/1667041500 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19c0036a0 con 0x7fb1a80a4ea0 2023-10-14T11:58:23.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.111+0000 7f9272ffd700 1 -- 172.21.15.78:0/3595420768 <== mon.2 v2:172.21.15.195:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f92740145a0 con 0x7f927c1ded30 2023-10-14T11:58:23.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.111+0000 7f11dad5c700 1 Processor -- start 2023-10-14T11:58:23.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.111+0000 7f6abaffd700 1 -- 172.21.15.78:0/181775399 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac0005d70 con 0x7f6ac406df80 2023-10-14T11:58:23.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.111+0000 7f6abaffd700 1 -- 172.21.15.78:0/181775399 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ac000ce30 con 0x7f6ac406df80 2023-10-14T11:58:23.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.111+0000 7f6abaffd700 1 -- 172.21.15.78:0/181775399 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac00072d0 con 0x7f6ac406df80 2023-10-14T11:58:23.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.111+0000 7f9272ffd700 1 --2- 172.21.15.78:0/3595420768 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f926806b090 0x7f926806d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.111+0000 7f9272ffd700 1 -- 172.21.15.78:0/3595420768 <== mon.2 v2:172.21.15.195:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9274006e40 con 0x7f927c1ded30 2023-10-14T11:58:23.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.112+0000 7f92820c0700 1 --2- 172.21.15.78:0/3595420768 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f926806b090 0x7f926806d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.112+0000 7fb1ad488700 1 -- 172.21.15.78:0/1667041500 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1a80a9860 con 0x7fb1a80a4ea0 2023-10-14T11:58:23.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.112+0000 7fb1ad488700 1 -- 172.21.15.78:0/1667041500 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1a80a9e20 con 0x7fb1a80a4ea0 2023-10-14T11:58:23.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.112+0000 7f92820c0700 1 --2- 172.21.15.78:0/3595420768 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f926806b090 0x7f926806d550 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f926c001740 tx=0x7f926c007a30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:23.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.111+0000 7f6acc3b5700 1 -- 172.21.15.78:0/181775399 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ac40bd550 con 0x7f6ac406df80 2023-10-14T11:58:23.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.111+0000 7f6acc3b5700 1 -- 172.21.15.78:0/181775399 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ac40bd270 con 0x7f6ac406df80 2023-10-14T11:58:23.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.112+0000 7f6acc3b5700 1 -- 172.21.15.78:0/181775399 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ac4082f40 con 0x7f6ac406df80 2023-10-14T11:58:23.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.113+0000 7f11dad5c700 1 -- start start 2023-10-14T11:58:23.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.114+0000 7fb18f7fe700 1 -- 172.21.15.78:0/1667041500 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb19c003800 con 0x7fb1a80a4ea0 2023-10-14T11:58:23.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.114+0000 7fb18f7fe700 1 --2- 172.21.15.78:0/1667041500 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb19406af00 0x7fb19406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.114+0000 7fb18f7fe700 1 -- 172.21.15.78:0/1667041500 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb19c08a3c0 con 0x7fb1a80a4ea0 2023-10-14T11:58:23.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.115+0000 7f11dad5c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f11d40a7ba0 0x7f11d40701f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.115+0000 7f11dad5c700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d40c5590 con 0x7f11d40c3550 2023-10-14T11:58:23.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.115+0000 7f11dad5c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d40c3fd0 con 0x7f11d40c1240 2023-10-14T11:58:23.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.115+0000 7f11dad5c700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11d406e040 con 0x7f11d40a7ba0 2023-10-14T11:58:23.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.115+0000 7f9284324700 1 -- 172.21.15.78:0/3595420768 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92640052f0 con 0x7f927c1ded30 2023-10-14T11:58:23.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.115+0000 7f11d92f9700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f11d40c3550 0x7f11d406d930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49746/0 (socket says 172.21.15.78:49746) 2023-10-14T11:58:23.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.115+0000 7f11d92f9700 1 -- 172.21.15.78:0/2249255421 learned_addr learned my addr 172.21.15.78:0/2249255421 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:23.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.115+0000 7fb1a6ffd700 1 --2- 172.21.15.78:0/1667041500 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb19406af00 0x7fb19406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.115+0000 7f11d8af8700 1 --2- 172.21.15.78:0/2249255421 >> v2:172.21.15.195:3300/0 conn(0x7f11d40a7ba0 0x7f11d40701f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.116+0000 7fb1a6ffd700 1 --2- 172.21.15.78:0/1667041500 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb19406af00 0x7fb19406d3c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb198009960 tx=0x7fb1980090e0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:23.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.116+0000 7f11d8af8700 1 -- 172.21.15.78:0/2249255421 >> v1:172.21.15.195:6789/0 conn(0x7f11d40c1240 legacy=0x7f11d406d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.116+0000 7f11d8af8700 1 -- 172.21.15.78:0/2249255421 >> v1:172.21.15.78:6789/0 conn(0x7f11d40c3550 legacy=0x7f11d406d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.116+0000 7f11d8af8700 1 -- 172.21.15.78:0/2249255421 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d40c4c80 con 0x7f11d40a7ba0 2023-10-14T11:58:23.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.116+0000 7f11d8af8700 1 --2- 172.21.15.78:0/2249255421 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f11d40a7ba0 0x7f11d40701f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f11c4009840 tx=0x7f11c4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa89b9f965fe41bf server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:23.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.116+0000 7f6abaffd700 1 -- 172.21.15.78:0/181775399 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6ac0007430 con 0x7f6ac406df80 2023-10-14T11:58:23.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.116+0000 7f11d37fe700 1 -- 172.21.15.78:0/2249255421 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c4011830 con 0x7f11d40a7ba0 2023-10-14T11:58:23.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.116+0000 7f11d37fe700 1 -- 172.21.15.78:0/2249255421 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c4011990 con 0x7f11d40a7ba0 2023-10-14T11:58:23.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.116+0000 7f11d37fe700 1 -- 172.21.15.78:0/2249255421 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c4011d20 con 0x7f11d40a7ba0 2023-10-14T11:58:23.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.113+0000 7f49ccff9700 1 -- 172.21.15.78:0/2393682697 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c00038a0 con 0x7f49d00c3fd0 2023-10-14T11:58:23.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.114+0000 7f49ccff9700 1 -- 172.21.15.78:0/2393682697 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c0003a20 con 0x7f49d00c3fd0 2023-10-14T11:58:23.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.114+0000 7f49ccff9700 1 -- 172.21.15.78:0/2393682697 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c0003dd0 con 0x7f49d00c3fd0 2023-10-14T11:58:23.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.117+0000 7f49d62c9700 1 -- 172.21.15.78:0/2393682697 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49d00ab930 con 0x7f49d00c3fd0 2023-10-14T11:58:23.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.117+0000 7f49d62c9700 1 -- 172.21.15.78:0/2393682697 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49d00ab250 con 0x7f49d00c3fd0 2023-10-14T11:58:23.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.117+0000 7fb1ad488700 1 -- 172.21.15.78:0/1667041500 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1a8074b80 con 0x7fb1a80a4ea0 2023-10-14T11:58:23.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.117+0000 7fe09dc91700 1 Processor -- start 2023-10-14T11:58:23.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.117+0000 7f11dad5c700 1 -- 172.21.15.78:0/2249255421 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f11d40a7ba0 msgr2=0x7f11d40701f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.117+0000 7f11dad5c700 1 --2- 172.21.15.78:0/2249255421 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f11d40a7ba0 0x7f11d40701f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f11c4009840 tx=0x7f11c4009b50 comp rx=0 tx=0).stop 2023-10-14T11:58:23.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.117+0000 7f6abaffd700 1 --2- 172.21.15.78:0/181775399 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f6ab006b0b0 0x7f6ab006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.117+0000 7f6abaffd700 1 -- 172.21.15.78:0/181775399 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6ac0088fe0 con 0x7f6ac406df80 2023-10-14T11:58:23.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.117+0000 7f11dad5c700 1 -- 172.21.15.78:0/2249255421 shutdown_connections 2023-10-14T11:58:23.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.117+0000 7f11dad5c700 1 --2- 172.21.15.78:0/2249255421 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f11d40a7ba0 0x7f11d40701f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.117+0000 7f11dad5c700 1 -- 172.21.15.78:0/2249255421 >> 172.21.15.78:0/2249255421 conn(0x7f11d406aab0 msgr2=0x7f11d406aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:23.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.117+0000 7fe09dc91700 1 -- start start 2023-10-14T11:58:23.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.118+0000 7fe09dc91700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe09806e3f0 0x7fe0980a6450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.118+0000 7fe09dc91700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe0980a8c20 0x7fe098163000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.118+0000 7fe09dc91700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe09806e8b0 con 0x7fe09806af60 2023-10-14T11:58:23.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.118+0000 7fe09dc91700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe098070a90 con 0x7fe0980a8c20 2023-10-14T11:58:23.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.118+0000 7fe09dc91700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe098070d70 con 0x7fe09806e3f0 2023-10-14T11:58:23.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.118+0000 7f11dad5c700 1 -- 172.21.15.78:0/2249255421 shutdown_connections 2023-10-14T11:58:23.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.118+0000 7f11dad5c700 1 -- 172.21.15.78:0/2249255421 wait complete. 2023-10-14T11:58:23.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.118+0000 7fe096ffd700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe09806e3f0 0x7fe0980a6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.118+0000 7fe096ffd700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe09806e3f0 0x7fe0980a6450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.78:44544/0 (socket says 172.21.15.78:44544) 2023-10-14T11:58:23.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.118+0000 7fe096ffd700 1 -- 172.21.15.78:0/3833874802 learned_addr learned my addr 172.21.15.78:0/3833874802 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:23.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.118+0000 7fe096ffd700 1 -- 172.21.15.78:0/3833874802 >> v2:172.21.15.195:3300/0 conn(0x7fe0980a8c20 msgr2=0x7fe098163000 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-14T11:58:23.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.118+0000 7fe096ffd700 1 --2- 172.21.15.78:0/3833874802 >> v2:172.21.15.195:3300/0 conn(0x7fe0980a8c20 0x7fe098163000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.118+0000 7fe096ffd700 1 -- 172.21.15.78:0/3833874802 >> v1:172.21.15.195:6789/0 conn(0x7fe09806af60 legacy=0x7fe098071850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.118+0000 7fe096ffd700 1 -- 172.21.15.78:0/3833874802 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe098071050 con 0x7fe09806e3f0 2023-10-14T11:58:23.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.118+0000 7fe096ffd700 1 --2- 172.21.15.78:0/3833874802 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe09806e3f0 0x7fe0980a6450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe090009e50 tx=0x7fe09000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42350ca1c8cd95b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:23.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.119+0000 7f11dad5c700 1 Processor -- start 2023-10-14T11:58:23.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.119+0000 7f11dad5c700 1 -- start start 2023-10-14T11:58:23.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.119+0000 7f86d12f4700 1 Processor -- start 2023-10-14T11:58:23.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.120+0000 7f6ac9950700 1 --2- 172.21.15.78:0/181775399 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f6ab006b0b0 0x7f6ab006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.120+0000 7f6ac9950700 1 --2- 172.21.15.78:0/181775399 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f6ab006b0b0 0x7f6ab006d570 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6ac406e3c0 tx=0x7f6ab4009f90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:23.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.120+0000 7f11dad5c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f11d40a7ba0 0x7f11d40bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.120+0000 7f11dad5c700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f11d40c3550 0x7f11d41576c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.120+0000 7f11dad5c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11d4155180 0x7f11d41ea790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.120+0000 7f11dad5c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d41ea370 con 0x7f11d4155180 2023-10-14T11:58:23.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.120+0000 7f11dad5c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d40bf8b0 con 0x7f11d40a7ba0 2023-10-14T11:58:23.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.120+0000 7f11dad5c700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11d40a4140 con 0x7f11d40c3550 2023-10-14T11:58:23.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.120+0000 7f11d92f9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11d4155180 0x7f11d41ea790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.120+0000 7f11d92f9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11d4155180 0x7f11d41ea790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44882/0 (socket says 172.21.15.78:44882) 2023-10-14T11:58:23.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.120+0000 7f11d92f9700 1 -- 172.21.15.78:0/1229261438 learned_addr learned my addr 172.21.15.78:0/1229261438 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:23.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.120+0000 7f11d8af8700 1 --2- 172.21.15.78:0/1229261438 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f11d40a7ba0 0x7f11d40bdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.121+0000 7f9272ffd700 1 -- 172.21.15.78:0/3595420768 <== mon.2 v2:172.21.15.195:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9274054140 con 0x7f927c1ded30 2023-10-14T11:58:23.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.120+0000 7f86d12f4700 1 -- start start 2023-10-14T11:58:23.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.120+0000 7f86d12f4700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f86cc06ab40 0x7f86cc0c1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.121+0000 7f86d12f4700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f86cc06e820 0x7f86cc14dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.121+0000 7f86d12f4700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86cc06ddd0 con 0x7f86cc0c3ec0 2023-10-14T11:58:23.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.121+0000 7f86d12f4700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86cc0c2620 con 0x7f86cc06e820 2023-10-14T11:58:23.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.121+0000 7f86d12f4700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86cc0c2900 con 0x7f86cc06ab40 2023-10-14T11:58:23.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.121+0000 7fe0967fc700 1 -- 172.21.15.78:0/3833874802 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09000e070 con 0x7fe09806e3f0 2023-10-14T11:58:23.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.121+0000 7fe0967fc700 1 -- 172.21.15.78:0/3833874802 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe09000b9c0 con 0x7fe09806e3f0 2023-10-14T11:58:23.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.121+0000 7fe0967fc700 1 -- 172.21.15.78:0/3833874802 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09000bd70 con 0x7fe09806e3f0 2023-10-14T11:58:23.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.121+0000 7f11d92f9700 1 -- 172.21.15.78:0/1229261438 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f11d40c3550 msgr2=0x7f11d41576c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-14T11:58:23.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.121+0000 7f11d92f9700 1 --2- 172.21.15.78:0/1229261438 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f11d40c3550 0x7f11d41576c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.121+0000 7f11d92f9700 1 -- 172.21.15.78:0/1229261438 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f11d40a7ba0 msgr2=0x7f11d40bdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.121+0000 7f11d92f9700 1 --2- 172.21.15.78:0/1229261438 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f11d40a7ba0 0x7f11d40bdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.121+0000 7f11d92f9700 1 -- 172.21.15.78:0/1229261438 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c40094a0 con 0x7f11d4155180 2023-10-14T11:58:23.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.121+0000 7f11d92f9700 1 --2- 172.21.15.78:0/1229261438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11d4155180 0x7f11d41ea790 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f11c800c400 tx=0x7f11c800c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:23.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.121+0000 7f86ca7fc700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f86cc0c3ec0 0x7f86cc150e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.78:39564/0 (socket says 172.21.15.78:39564) 2023-10-14T11:58:23.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.121+0000 7f86ca7fc700 1 -- 172.21.15.78:0/565284889 learned_addr learned my addr 172.21.15.78:0/565284889 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:23.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.121+0000 7f86c9ffb700 1 -- 172.21.15.78:0/565284889 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2023700341 0 0) 0x7f86cc06ddd0 con 0x7f86cc0c3ec0 2023-10-14T11:58:23.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.121+0000 7f86c9ffb700 1 -- 172.21.15.78:0/565284889 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86b4003590 con 0x7f86cc0c3ec0 2023-10-14T11:58:23.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.121+0000 7f86c9ffb700 1 -- 172.21.15.78:0/565284889 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 445793280 0 0) 0x7f86b4003590 con 0x7f86cc0c3ec0 2023-10-14T11:58:23.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.121+0000 7f86c9ffb700 1 -- 172.21.15.78:0/565284889 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f86cc06ab40 msgr2=0x7f86cc0c1dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-14T11:58:23.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.121+0000 7f86c9ffb700 1 --2- 172.21.15.78:0/565284889 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f86cc06ab40 0x7f86cc0c1dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.121+0000 7f86c9ffb700 1 -- 172.21.15.78:0/565284889 >> v2:172.21.15.78:3300/0 conn(0x7f86cc06e820 msgr2=0x7f86cc14dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:23.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.121+0000 7f86c9ffb700 1 --2- 172.21.15.78:0/565284889 >> v2:172.21.15.78:3300/0 conn(0x7f86cc06e820 0x7f86cc14dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.121+0000 7f86c9ffb700 1 -- 172.21.15.78:0/565284889 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86cc06e020 con 0x7f86cc0c3ec0 2023-10-14T11:58:23.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.121+0000 7f86c9ffb700 1 -- 172.21.15.78:0/565284889 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441799917 0 0) 0x7f86c0001670 con 0x7f86cc0c3ec0 2023-10-14T11:58:23.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.121+0000 7f86c9ffb700 1 -- 172.21.15.78:0/565284889 >> v1:172.21.15.195:6789/0 conn(0x7f86cc0c3ec0 legacy=0x7f86cc150e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.121+0000 7f86c9ffb700 1 --2- 172.21.15.78:0/565284889 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86b4003bb0 0x7f86b4003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.122+0000 7f86c9ffb700 1 --2- 172.21.15.78:0/565284889 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f86b40067f0 0x7f86b4008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.122+0000 7f86c9ffb700 1 --2- 172.21.15.78:0/565284889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86b40094d0 0x7f86b4009950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.122+0000 7f86c9ffb700 1 -- 172.21.15.78:0/565284889 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86cc0c2620 con 0x7f86b40094d0 2023-10-14T11:58:23.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.122+0000 7f86c9ffb700 1 -- 172.21.15.78:0/565284889 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86cc0c2900 con 0x7f86b4003bb0 2023-10-14T11:58:23.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.122+0000 7f86c9ffb700 1 -- 172.21.15.78:0/565284889 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86b400a2b0 con 0x7f86b40067f0 2023-10-14T11:58:23.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.122+0000 7f49ccff9700 1 -- 172.21.15.78:0/2393682697 <== mon.2 v2:172.21.15.195:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f49c0017420 con 0x7f49d00c3fd0 2023-10-14T11:58:23.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.122+0000 7fe09dc91700 1 -- 172.21.15.78:0/3833874802 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe09806e3f0 msgr2=0x7fe0980a6450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.122+0000 7fe09dc91700 1 --2- 172.21.15.78:0/3833874802 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe09806e3f0 0x7fe0980a6450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe090009e50 tx=0x7fe09000b330 comp rx=0 tx=0).stop 2023-10-14T11:58:23.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.122+0000 7fe09dc91700 1 -- 172.21.15.78:0/3833874802 shutdown_connections 2023-10-14T11:58:23.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.122+0000 7fe09dc91700 1 --2- 172.21.15.78:0/3833874802 >> v2:172.21.15.195:3300/0 conn(0x7fe0980a8c20 0x7fe098163000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.122+0000 7fe09dc91700 1 --2- 172.21.15.78:0/3833874802 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe09806e3f0 0x7fe0980a6450 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.193 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.122+0000 7fe09dc91700 1 -- 172.21.15.78:0/3833874802 >> 172.21.15.78:0/3833874802 conn(0x7fe0980c6b10 msgr2=0x7fe0980c6f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:23.193 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.122+0000 7f86ca7fc700 1 --2- 172.21.15.78:0/565284889 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f86b40067f0 0x7f86b4008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.193 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.122+0000 7f86cb7fe700 1 --2- 172.21.15.78:0/565284889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86b40094d0 0x7f86b4009950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.122+0000 7fe09dc91700 1 -- 172.21.15.78:0/3833874802 shutdown_connections 2023-10-14T11:58:23.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.122+0000 7f86ca7fc700 1 -- 172.21.15.78:0/565284889 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86b4003bb0 msgr2=0x7f86b4003fd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-14T11:58:23.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.122+0000 7f86ca7fc700 1 --2- 172.21.15.78:0/565284889 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86b4003bb0 0x7f86b4003fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.122+0000 7f86ca7fc700 1 -- 172.21.15.78:0/565284889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86b40094d0 msgr2=0x7f86b4009950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:23.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.122+0000 7f86ca7fc700 1 --2- 172.21.15.78:0/565284889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86b40094d0 0x7f86b4009950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.122+0000 7f86ca7fc700 1 -- 172.21.15.78:0/565284889 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b4003590 con 0x7f86b40067f0 2023-10-14T11:58:23.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.122+0000 7f86cb7fe700 1 --2- 172.21.15.78:0/565284889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86b40094d0 0x7f86b4009950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T11:58:23.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.122+0000 7f49ccff9700 1 --2- 172.21.15.78:0/2393682697 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f49bc06af50 0x7f49bc06d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.122+0000 7f86ca7fc700 1 --2- 172.21.15.78:0/565284889 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f86b40067f0 0x7f86b4008c80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f86c000c320 tx=0x7f86b4003840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=533f727307e94a3b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:23.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.122+0000 7f49cffff700 1 --2- 172.21.15.78:0/2393682697 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f49bc06af50 0x7f49bc06d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.122+0000 7f49ccff9700 1 -- 172.21.15.78:0/2393682697 <== mon.2 v2:172.21.15.195:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f49c0013070 con 0x7f49d00c3fd0 2023-10-14T11:58:23.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.123+0000 7f86c9ffb700 1 -- 172.21.15.78:0/565284889 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c000ce60 con 0x7f86b40067f0 2023-10-14T11:58:23.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.123+0000 7f86c9ffb700 1 -- 172.21.15.78:0/565284889 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c0005960 con 0x7f86b40067f0 2023-10-14T11:58:23.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.123+0000 7f86c9ffb700 1 -- 172.21.15.78:0/565284889 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c0005d10 con 0x7f86b40067f0 2023-10-14T11:58:23.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.123+0000 7f49cffff700 1 --2- 172.21.15.78:0/2393682697 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f49bc06af50 0x7f49bc06d410 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f49c8005c90 tx=0x7f49c8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:23.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.122+0000 7fe09dc91700 1 -- 172.21.15.78:0/3833874802 wait complete. 2023-10-14T11:58:23.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.123+0000 7fe09dc91700 1 Processor -- start 2023-10-14T11:58:23.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.124+0000 7f86d12f4700 1 -- 172.21.15.78:0/565284889 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f86b40067f0 msgr2=0x7f86b4008c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.124+0000 7f86d12f4700 1 --2- 172.21.15.78:0/565284889 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f86b40067f0 0x7f86b4008c80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f86c000c320 tx=0x7f86b4003840 comp rx=0 tx=0).stop 2023-10-14T11:58:23.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.124+0000 7f86cb7fe700 1 -- 172.21.15.78:0/565284889 reap_dead start 2023-10-14T11:58:23.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.124+0000 7f86d12f4700 1 -- 172.21.15.78:0/565284889 shutdown_connections 2023-10-14T11:58:23.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.124+0000 7f86d12f4700 1 -- 172.21.15.78:0/565284889 >> 172.21.15.78:0/565284889 conn(0x7f86cc0a84a0 msgr2=0x7f86cc0a88b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:23.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.124+0000 7f6abaffd700 1 -- 172.21.15.78:0/181775399 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6ac0054230 con 0x7f6ac406df80 2023-10-14T11:58:23.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.124+0000 7fe09dc91700 1 -- start start 2023-10-14T11:58:23.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.124+0000 7fe09dc91700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe09806af60 0x7fe098154630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.124+0000 7fe09dc91700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe09806e3f0 0x7fe098141950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.125+0000 7fe09dc91700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe0980a8c20 0x7fe098142f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.125+0000 7fe09dc91700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0981fb020 con 0x7fe09806e3f0 2023-10-14T11:58:23.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.125+0000 7fe09dc91700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0981fa410 con 0x7fe09806af60 2023-10-14T11:58:23.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.125+0000 7fe09dc91700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0981fa6f0 con 0x7fe0980a8c20 2023-10-14T11:58:23.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.125+0000 7f86d12f4700 1 -- 172.21.15.78:0/565284889 shutdown_connections 2023-10-14T11:58:23.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.125+0000 7fe0977fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe09806af60 0x7fe098154630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.125+0000 7fe0977fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe09806af60 0x7fe098154630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.78:47636/0 (socket says 172.21.15.78:47636) 2023-10-14T11:58:23.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.125+0000 7fe0977fe700 1 -- 172.21.15.78:0/1474353595 learned_addr learned my addr 172.21.15.78:0/1474353595 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:23.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.125+0000 7fe096ffd700 1 --2- 172.21.15.78:0/1474353595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe09806e3f0 0x7fe098141950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.125+0000 7f86d12f4700 1 -- 172.21.15.78:0/565284889 wait complete. 2023-10-14T11:58:23.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.125+0000 7fe0977fe700 1 -- 172.21.15.78:0/1474353595 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe0980a8c20 msgr2=0x7fe098142f50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-14T11:58:23.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.125+0000 7fe0977fe700 1 --2- 172.21.15.78:0/1474353595 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe0980a8c20 0x7fe098142f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.125+0000 7fe0977fe700 1 -- 172.21.15.78:0/1474353595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe09806e3f0 msgr2=0x7fe098141950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.125+0000 7fe0977fe700 1 --2- 172.21.15.78:0/1474353595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe09806e3f0 0x7fe098141950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.125+0000 7fe0977fe700 1 -- 172.21.15.78:0/1474353595 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe090009b30 con 0x7fe09806af60 2023-10-14T11:58:23.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.125+0000 7fe0977fe700 1 --2- 172.21.15.78:0/1474353595 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe09806af60 0x7fe098154630 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe08c00c4e0 tx=0x7fe08c00c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:23.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.125+0000 7fe07ffff700 1 -- 172.21.15.78:0/1474353595 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c0052d0 con 0x7fe09806af60 2023-10-14T11:58:23.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.125+0000 7fe07ffff700 1 -- 172.21.15.78:0/1474353595 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08c005430 con 0x7fe09806af60 2023-10-14T11:58:23.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.125+0000 7fe09dc91700 1 -- 172.21.15.78:0/1474353595 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0981fa9d0 con 0x7fe09806af60 2023-10-14T11:58:23.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.126+0000 7fe07ffff700 1 -- 172.21.15.78:0/1474353595 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c0038c0 con 0x7fe09806af60 2023-10-14T11:58:23.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.126+0000 7fe09dc91700 1 -- 172.21.15.78:0/1474353595 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe098141e90 con 0x7fe09806af60 2023-10-14T11:58:23.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.127+0000 7f86d12f4700 1 Processor -- start 2023-10-14T11:58:23.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.127+0000 7fe07ffff700 1 -- 172.21.15.78:0/1474353595 <== mon.1 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe08c003a20 con 0x7fe09806af60 2023-10-14T11:58:23.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.127+0000 7fe07ffff700 1 --2- 172.21.15.78:0/1474353595 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe08406af00 0x7fe08406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.128+0000 7fe07ffff700 1 -- 172.21.15.78:0/1474353595 <== mon.1 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe08c088a80 con 0x7fe09806af60 2023-10-14T11:58:23.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.128+0000 7fe096ffd700 1 --2- 172.21.15.78:0/1474353595 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe08406af00 0x7fe08406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.127+0000 7f86d12f4700 1 -- start start 2023-10-14T11:58:23.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.128+0000 7f86d12f4700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86b40067f0 0x7f86cc149c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.128+0000 7f86d12f4700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f86cc154320 0x7f86cc147ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.128+0000 7f86d12f4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86cc0bbda0 0x7f86cc0ba3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.128+0000 7f86d12f4700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86cc148700 con 0x7f86cc0bbda0 2023-10-14T11:58:23.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.128+0000 7f86d12f4700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86cc149770 con 0x7f86b40067f0 2023-10-14T11:58:23.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.128+0000 7f86d12f4700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86cc0bc8f0 con 0x7f86cc154320 2023-10-14T11:58:23.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.128+0000 7f86cb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86cc0bbda0 0x7f86cc0ba3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.128+0000 7f86cb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86cc0bbda0 0x7f86cc0ba3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44906/0 (socket says 172.21.15.78:44906) 2023-10-14T11:58:23.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.128+0000 7f86cb7fe700 1 -- 172.21.15.78:0/4142975443 learned_addr learned my addr 172.21.15.78:0/4142975443 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:23.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.128+0000 7f86caffd700 1 --2- 172.21.15.78:0/4142975443 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86b40067f0 0x7f86cc149c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.128+0000 7f11d17fa700 1 -- 172.21.15.78:0/1229261438 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c8005d70 con 0x7f11d4155180 2023-10-14T11:58:23.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.128+0000 7f11d17fa700 1 -- 172.21.15.78:0/1229261438 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c800ce80 con 0x7f11d4155180 2023-10-14T11:58:23.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.128+0000 7f11d17fa700 1 -- 172.21.15.78:0/1229261438 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c80072d0 con 0x7f11d4155180 2023-10-14T11:58:23.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.128+0000 7fb18f7fe700 1 -- 172.21.15.78:0/1667041500 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb19c0553e0 con 0x7fb1a80a4ea0 2023-10-14T11:58:23.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.128+0000 7fe096ffd700 1 --2- 172.21.15.78:0/1474353595 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe08406af00 0x7fe08406d3c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe09000b370 tx=0x7fe090009000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:23.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.129+0000 7f49d62c9700 1 -- 172.21.15.78:0/2393682697 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49d0074b80 con 0x7f49d00c3fd0 2023-10-14T11:58:23.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.128+0000 7f86ca7fc700 1 --2- 172.21.15.78:0/4142975443 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f86cc154320 0x7f86cc147ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.129+0000 7f86caffd700 1 -- 172.21.15.78:0/4142975443 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f86cc154320 msgr2=0x7f86cc147ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.129+0000 7f86caffd700 1 --2- 172.21.15.78:0/4142975443 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f86cc154320 0x7f86cc147ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.129+0000 7f86caffd700 1 -- 172.21.15.78:0/4142975443 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86cc0bbda0 msgr2=0x7f86cc0ba3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.129+0000 7f86caffd700 1 --2- 172.21.15.78:0/4142975443 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86cc0bbda0 0x7f86cc0ba3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.129+0000 7f86caffd700 1 -- 172.21.15.78:0/4142975443 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86cc148420 con 0x7f86b40067f0 2023-10-14T11:58:23.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.129+0000 7f86caffd700 1 --2- 172.21.15.78:0/4142975443 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86b40067f0 0x7f86cc149c90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f86cc071210 tx=0x7f86bc00b670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:23.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.129+0000 7f86b37fe700 1 -- 172.21.15.78:0/4142975443 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86bc011620 con 0x7f86b40067f0 2023-10-14T11:58:23.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.129+0000 7f86b37fe700 1 -- 172.21.15.78:0/4142975443 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86bc0117a0 con 0x7f86b40067f0 2023-10-14T11:58:23.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.129+0000 7f86b37fe700 1 -- 172.21.15.78:0/4142975443 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86bc011b50 con 0x7f86b40067f0 2023-10-14T11:58:23.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.129+0000 7f86d12f4700 1 -- 172.21.15.78:0/4142975443 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86cc0b9ea0 con 0x7f86b40067f0 2023-10-14T11:58:23.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.129+0000 7f86d12f4700 1 -- 172.21.15.78:0/4142975443 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86cc151080 con 0x7f86b40067f0 2023-10-14T11:58:23.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.130+0000 7fe09dc91700 1 -- 172.21.15.78:0/1474353595 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0780052f0 con 0x7fe09806af60 2023-10-14T11:58:23.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.131+0000 7f86b37fe700 1 -- 172.21.15.78:0/4142975443 <== mon.1 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f86bc011cb0 con 0x7f86b40067f0 2023-10-14T11:58:23.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.130+0000 7f11dad5c700 1 -- 172.21.15.78:0/1229261438 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11d40bfb00 con 0x7f11d4155180 2023-10-14T11:58:23.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.130+0000 7f11dad5c700 1 -- 172.21.15.78:0/1229261438 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11d40c00c0 con 0x7f11d4155180 2023-10-14T11:58:23.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.131+0000 7f11dad5c700 1 -- 172.21.15.78:0/1229261438 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11d41ea4b0 con 0x7f11d4155180 2023-10-14T11:58:23.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.132+0000 7f86b37fe700 1 --2- 172.21.15.78:0/4142975443 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f86b806af30 0x7f86b806d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.132+0000 7f86b37fe700 1 -- 172.21.15.78:0/4142975443 <== mon.1 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f86bc088dd0 con 0x7f86b40067f0 2023-10-14T11:58:23.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.133+0000 7f86d12f4700 1 -- 172.21.15.78:0/4142975443 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86cc0b9d00 con 0x7f86b40067f0 2023-10-14T11:58:23.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.134+0000 7f11d17fa700 1 -- 172.21.15.78:0/1229261438 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f11c8007430 con 0x7f11d4155180 2023-10-14T11:58:23.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.134+0000 7f11d17fa700 1 --2- 172.21.15.78:0/1229261438 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f11c006af50 0x7f11c006d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.134+0000 7f11d8af8700 1 --2- 172.21.15.78:0/1229261438 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f11c006af50 0x7f11c006d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.134+0000 7f11d17fa700 1 -- 172.21.15.78:0/1229261438 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f11c8088e90 con 0x7f11d4155180 2023-10-14T11:58:23.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.135+0000 7f86ca7fc700 1 --2- 172.21.15.78:0/4142975443 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f86b806af30 0x7f86b806d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.135+0000 7f11d8af8700 1 --2- 172.21.15.78:0/1229261438 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f11c006af50 0x7f11c006d410 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f11c4000c00 tx=0x7f11c4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:23.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.135+0000 7f86ca7fc700 1 --2- 172.21.15.78:0/4142975443 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f86b806af30 0x7f86b806d3f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f86c0000c00 tx=0x7f86c000a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:23.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.135+0000 7fe07ffff700 1 -- 172.21.15.78:0/1474353595 <== mon.1 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe08c053b40 con 0x7fe09806af60 2023-10-14T11:58:23.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.138+0000 7f49ccff9700 1 -- 172.21.15.78:0/2393682697 <== mon.2 v2:172.21.15.195:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f49c0056480 con 0x7f49d00c3fd0 2023-10-14T11:58:23.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.141+0000 7f11d17fa700 1 -- 172.21.15.78:0/1229261438 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f11c8054060 con 0x7f11d4155180 2023-10-14T11:58:23.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.147+0000 7f86b37fe700 1 -- 172.21.15.78:0/4142975443 <== mon.1 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f86cc0b9d00 con 0x7f86b40067f0 2023-10-14T11:58:23.232 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T11:58:23.273 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T11:58:23.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.476+0000 7fcc7c665700 1 -- 172.21.15.78:0/757927800 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fcc5c0055a0 con 0x7fcc740c2880 2023-10-14T11:58:23.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.493+0000 7fcc6affd700 1 -- 172.21.15.78:0/757927800 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcc640579b0 con 0x7fcc740c2880 2023-10-14T11:58:23.498 INFO:teuthology.orchestra.run.smithi078.stdout:55834574853 2023-10-14T11:58:23.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.502+0000 7fcc7c665700 1 -- 172.21.15.78:0/757927800 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fcc5806afa0 msgr2=0x7fcc5806d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.502+0000 7fcc7c665700 1 --2- 172.21.15.78:0/757927800 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fcc5806afa0 0x7fcc5806d460 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c009c60 tx=0x7fcc6c009040 comp rx=0 tx=0).stop 2023-10-14T11:58:23.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.502+0000 7fcc7c665700 1 -- 172.21.15.78:0/757927800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc740c2880 msgr2=0x7fcc740bd8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.502+0000 7fcc7c665700 1 --2- 172.21.15.78:0/757927800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc740c2880 0x7fcc740bd8e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fcc6400c450 tx=0x7fcc6400c760 comp rx=0 tx=0).stop 2023-10-14T11:58:23.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.503+0000 7fcc7c665700 1 -- 172.21.15.78:0/757927800 shutdown_connections 2023-10-14T11:58:23.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.503+0000 7fcc7c665700 1 --2- 172.21.15.78:0/757927800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc740c2880 0x7fcc740bd8e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.503+0000 7fcc7c665700 1 --2- 172.21.15.78:0/757927800 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fcc5806afa0 0x7fcc5806d460 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.503+0000 7fcc7c665700 1 --2- 172.21.15.78:0/757927800 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcc740706b0 0x7fcc740be600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.503+0000 7fcc7c665700 1 --2- 172.21.15.78:0/757927800 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcc740c0830 0x7fcc740bd3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.503+0000 7fcc7c665700 1 -- 172.21.15.78:0/757927800 >> 172.21.15.78:0/757927800 conn(0x7fcc7406aab0 msgr2=0x7fcc74152150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:23.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.503+0000 7fcc7c665700 1 -- 172.21.15.78:0/757927800 shutdown_connections 2023-10-14T11:58:23.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.503+0000 7fcc7c665700 1 -- 172.21.15.78:0/757927800 wait complete. 2023-10-14T11:58:23.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.513+0000 7fb1ad488700 1 -- 172.21.15.78:0/1667041500 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fb1a80abe30 con 0x7fb1a80a4ea0 2023-10-14T11:58:23.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.513+0000 7fb18f7fe700 1 -- 172.21.15.78:0/1667041500 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb19c01b020 con 0x7fb1a80a4ea0 2023-10-14T11:58:23.518 INFO:teuthology.orchestra.run.smithi078.stdout:55834574854 2023-10-14T11:58:23.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.526+0000 7f6acc3b5700 1 -- 172.21.15.78:0/181775399 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f6ac4070a30 con 0x7f6ac406df80 2023-10-14T11:58:23.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.527+0000 7f6abaffd700 1 -- 172.21.15.78:0/181775399 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6ac0021020 con 0x7f6ac406df80 2023-10-14T11:58:23.529 INFO:teuthology.orchestra.run.smithi078.stdout:55834574854 2023-10-14T11:58:23.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.528+0000 7fb1ad488700 1 -- 172.21.15.78:0/1667041500 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb19406af00 msgr2=0x7fb19406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.528+0000 7fb1ad488700 1 --2- 172.21.15.78:0/1667041500 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb19406af00 0x7fb19406d3c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb198009960 tx=0x7fb1980090e0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.528+0000 7fb1ad488700 1 -- 172.21.15.78:0/1667041500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1a80a4ea0 msgr2=0x7fb1a8200930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.528+0000 7fb1ad488700 1 --2- 172.21.15.78:0/1667041500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1a80a4ea0 0x7fb1a8200930 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb19c00d370 tx=0x7fb19c004750 comp rx=0 tx=0).stop 2023-10-14T11:58:23.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.528+0000 7f9284324700 1 -- 172.21.15.78:0/3595420768 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f9264005870 con 0x7f927c1ded30 2023-10-14T11:58:23.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.529+0000 7fb1ad488700 1 -- 172.21.15.78:0/1667041500 shutdown_connections 2023-10-14T11:58:23.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.529+0000 7fb1ad488700 1 --2- 172.21.15.78:0/1667041500 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb1a80c2ed0 0x7fb1a81fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.529+0000 7fb1ad488700 1 --2- 172.21.15.78:0/1667041500 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb19406af00 0x7fb19406d3c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.529+0000 7fb1ad488700 1 --2- 172.21.15.78:0/1667041500 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb1a806df80 0x7fb1a82003f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.529+0000 7fb1ad488700 1 --2- 172.21.15.78:0/1667041500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1a80a4ea0 0x7fb1a8200930 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.529+0000 7fb1ad488700 1 -- 172.21.15.78:0/1667041500 >> 172.21.15.78:0/1667041500 conn(0x7fb1a806aab0 msgr2=0x7fb1a8071bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:23.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.529+0000 7fb1ad488700 1 -- 172.21.15.78:0/1667041500 shutdown_connections 2023-10-14T11:58:23.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.529+0000 7fb1ad488700 1 -- 172.21.15.78:0/1667041500 wait complete. 2023-10-14T11:58:23.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.531+0000 7f9272ffd700 1 -- 172.21.15.78:0/3595420768 <== mon.2 v2:172.21.15.195:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9274019070 con 0x7f927c1ded30 2023-10-14T11:58:23.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.531+0000 7f11dad5c700 1 -- 172.21.15.78:0/1229261438 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f11d41ea4b0 con 0x7f11d4155180 2023-10-14T11:58:23.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.532+0000 7f11d17fa700 1 -- 172.21.15.78:0/1229261438 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f11c8021020 con 0x7f11d4155180 2023-10-14T11:58:23.535 INFO:teuthology.orchestra.run.smithi078.stdout:55834574854 2023-10-14T11:58:23.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.532+0000 7f6acc3b5700 1 -- 172.21.15.78:0/181775399 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f6ab006b0b0 msgr2=0x7f6ab006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.532+0000 7f6acc3b5700 1 --2- 172.21.15.78:0/181775399 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f6ab006b0b0 0x7f6ab006d570 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6ac406e3c0 tx=0x7f6ab4009f90 comp rx=0 tx=0).stop 2023-10-14T11:58:23.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.532+0000 7f6acc3b5700 1 -- 172.21.15.78:0/181775399 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ac406df80 msgr2=0x7f6ac4200000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.532+0000 7f6acc3b5700 1 --2- 172.21.15.78:0/181775399 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ac406df80 0x7f6ac4200000 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6ac000c4c0 tx=0x7f6ac000c7d0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.532+0000 7f6acc3b5700 1 -- 172.21.15.78:0/181775399 shutdown_connections 2023-10-14T11:58:23.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.532+0000 7f6acc3b5700 1 --2- 172.21.15.78:0/181775399 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ac40c2ed0 0x7f6ac4085410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.532+0000 7f6acc3b5700 1 --2- 172.21.15.78:0/181775399 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ac406df80 0x7f6ac4200000 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.532+0000 7f6acc3b5700 1 --2- 172.21.15.78:0/181775399 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f6ab006b0b0 0x7f6ab006d570 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.532+0000 7f6acc3b5700 1 --2- 172.21.15.78:0/181775399 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ac40a4ea0 0x7f6ac40830f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.532+0000 7f6acc3b5700 1 -- 172.21.15.78:0/181775399 >> 172.21.15.78:0/181775399 conn(0x7f6ac406aab0 msgr2=0x7f6ac40c53d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:23.538 INFO:teuthology.orchestra.run.smithi078.stdout:55834574853 2023-10-14T11:58:23.539 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574853 for osd.0 2023-10-14T11:58:23.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.534+0000 7f6acc3b5700 1 -- 172.21.15.78:0/181775399 shutdown_connections 2023-10-14T11:58:23.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.534+0000 7f6acc3b5700 1 -- 172.21.15.78:0/181775399 wait complete. 2023-10-14T11:58:23.540 DEBUG:teuthology.parallel:result is None 2023-10-14T11:58:23.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.536+0000 7f11dad5c700 1 -- 172.21.15.78:0/1229261438 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f11c006af50 msgr2=0x7f11c006d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.536+0000 7f11dad5c700 1 --2- 172.21.15.78:0/1229261438 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f11c006af50 0x7f11c006d410 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f11c4000c00 tx=0x7f11c4008030 comp rx=0 tx=0).stop 2023-10-14T11:58:23.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.536+0000 7f11dad5c700 1 -- 172.21.15.78:0/1229261438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11d4155180 msgr2=0x7f11d41ea790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.536+0000 7f11dad5c700 1 --2- 172.21.15.78:0/1229261438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11d4155180 0x7f11d41ea790 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f11c800c400 tx=0x7f11c800c710 comp rx=0 tx=0).stop 2023-10-14T11:58:23.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.536+0000 7f9284324700 1 -- 172.21.15.78:0/3595420768 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f926806b090 msgr2=0x7f926806d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.536+0000 7f9284324700 1 --2- 172.21.15.78:0/3595420768 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f926806b090 0x7f926806d550 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f926c001740 tx=0x7f926c007a30 comp rx=0 tx=0).stop 2023-10-14T11:58:23.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.536+0000 7f9284324700 1 -- 172.21.15.78:0/3595420768 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f927c1ded30 msgr2=0x7f927c1f1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.536+0000 7f9284324700 1 --2- 172.21.15.78:0/3595420768 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f927c1ded30 0x7f927c1f1b90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f927400ba10 tx=0x7f927400bdd0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.537+0000 7f11dad5c700 1 -- 172.21.15.78:0/1229261438 shutdown_connections 2023-10-14T11:58:23.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.537+0000 7f11dad5c700 1 --2- 172.21.15.78:0/1229261438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11d4155180 0x7f11d41ea790 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.537+0000 7f11dad5c700 1 --2- 172.21.15.78:0/1229261438 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f11c006af50 0x7f11c006d410 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.537+0000 7f11dad5c700 1 --2- 172.21.15.78:0/1229261438 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f11d40a7ba0 0x7f11d40bdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.537+0000 7f11dad5c700 1 --2- 172.21.15.78:0/1229261438 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f11d40c3550 0x7f11d41576c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.537+0000 7f11dad5c700 1 -- 172.21.15.78:0/1229261438 >> 172.21.15.78:0/1229261438 conn(0x7f11d406aab0 msgr2=0x7f11d40c07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:23.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.537+0000 7fe09dc91700 1 -- 172.21.15.78:0/1474353595 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fe078005870 con 0x7fe09806af60 2023-10-14T11:58:23.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.538+0000 7f11dad5c700 1 -- 172.21.15.78:0/1229261438 shutdown_connections 2023-10-14T11:58:23.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.538+0000 7f11dad5c700 1 -- 172.21.15.78:0/1229261438 wait complete. 2023-10-14T11:58:23.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.538+0000 7fe07ffff700 1 -- 172.21.15.78:0/1474353595 <== mon.1 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe08c017070 con 0x7fe09806af60 2023-10-14T11:58:23.546 INFO:teuthology.orchestra.run.smithi078.stdout:55834574854 2023-10-14T11:58:23.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.539+0000 7f9284324700 1 -- 172.21.15.78:0/3595420768 shutdown_connections 2023-10-14T11:58:23.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.539+0000 7f9284324700 1 --2- 172.21.15.78:0/3595420768 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f927c1ded30 0x7f927c1f1b90 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.539+0000 7f9284324700 1 --2- 172.21.15.78:0/3595420768 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f926806b090 0x7f926806d550 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.539+0000 7f9284324700 1 --2- 172.21.15.78:0/3595420768 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f927c072910 0x7f927c1e0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.539+0000 7f9284324700 1 --2- 172.21.15.78:0/3595420768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f927c0c65c0 0x7f927c1f1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.539+0000 7f9284324700 1 -- 172.21.15.78:0/3595420768 >> 172.21.15.78:0/3595420768 conn(0x7f927c06af90 msgr2=0x7f927c0c4470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:23.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.542+0000 7fe09dc91700 1 -- 172.21.15.78:0/1474353595 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe08406af00 msgr2=0x7fe08406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.542+0000 7fe09dc91700 1 --2- 172.21.15.78:0/1474353595 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe08406af00 0x7fe08406d3c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe09000b370 tx=0x7fe090009000 comp rx=0 tx=0).stop 2023-10-14T11:58:23.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.542+0000 7fe09dc91700 1 -- 172.21.15.78:0/1474353595 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe09806af60 msgr2=0x7fe098154630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.542+0000 7fe09dc91700 1 --2- 172.21.15.78:0/1474353595 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe09806af60 0x7fe098154630 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe08c00c4e0 tx=0x7fe08c00c8a0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.543+0000 7f9284324700 1 -- 172.21.15.78:0/3595420768 shutdown_connections 2023-10-14T11:58:23.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.543+0000 7f9284324700 1 -- 172.21.15.78:0/3595420768 wait complete. 2023-10-14T11:58:23.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.547+0000 7f49d62c9700 1 -- 172.21.15.78:0/2393682697 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f49d00a4a00 con 0x7f49d00c3fd0 2023-10-14T11:58:23.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.548+0000 7f49ccff9700 1 -- 172.21.15.78:0/2393682697 <== mon.2 v2:172.21.15.195:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f49c0025030 con 0x7f49d00c3fd0 2023-10-14T11:58:23.551 INFO:teuthology.orchestra.run.smithi078.stdout:55834574854 2023-10-14T11:58:23.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.549+0000 7fe09dc91700 1 -- 172.21.15.78:0/1474353595 shutdown_connections 2023-10-14T11:58:23.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.549+0000 7fe09dc91700 1 --2- 172.21.15.78:0/1474353595 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe0980a8c20 0x7fe098142f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.549+0000 7fe09dc91700 1 --2- 172.21.15.78:0/1474353595 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe08406af00 0x7fe08406d3c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.549+0000 7fe09dc91700 1 --2- 172.21.15.78:0/1474353595 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe09806af60 0x7fe098154630 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.549+0000 7fe09dc91700 1 --2- 172.21.15.78:0/1474353595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe09806e3f0 0x7fe098141950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.549+0000 7fe09dc91700 1 -- 172.21.15.78:0/1474353595 >> 172.21.15.78:0/1474353595 conn(0x7fe0980c6b10 msgr2=0x7fe0980a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:23.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.550+0000 7f86d12f4700 1 -- 172.21.15.78:0/4142975443 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f86cc0b9d00 con 0x7f86b40067f0 2023-10-14T11:58:23.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.550+0000 7fe09dc91700 1 -- 172.21.15.78:0/1474353595 shutdown_connections 2023-10-14T11:58:23.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.550+0000 7fe09dc91700 1 -- 172.21.15.78:0/1474353595 wait complete. 2023-10-14T11:58:23.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.550+0000 7f86b37fe700 1 -- 172.21.15.78:0/4142975443 <== mon.1 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f86cc0b9d00 con 0x7f86b40067f0 2023-10-14T11:58:23.555 INFO:teuthology.orchestra.run.smithi078.stdout:55834574854 2023-10-14T11:58:23.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.551+0000 7f49d62c9700 1 -- 172.21.15.78:0/2393682697 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f49bc06af50 msgr2=0x7f49bc06d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.551+0000 7f49d62c9700 1 --2- 172.21.15.78:0/2393682697 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f49bc06af50 0x7f49bc06d410 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f49c8005c90 tx=0x7f49c8009040 comp rx=0 tx=0).stop 2023-10-14T11:58:23.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.551+0000 7f49d62c9700 1 -- 172.21.15.78:0/2393682697 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f49d00c3fd0 msgr2=0x7f49d00893a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.551+0000 7f49d62c9700 1 --2- 172.21.15.78:0/2393682697 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f49d00c3fd0 0x7f49d00893a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f49c0018000 tx=0x7f49c0004750 comp rx=0 tx=0).stop 2023-10-14T11:58:23.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.552+0000 7f49d62c9700 1 -- 172.21.15.78:0/2393682697 shutdown_connections 2023-10-14T11:58:23.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.552+0000 7f49d62c9700 1 --2- 172.21.15.78:0/2393682697 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f49bc06af50 0x7f49bc06d410 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.552+0000 7f49d62c9700 1 --2- 172.21.15.78:0/2393682697 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f49d00c4970 0x7f49d0088810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.552+0000 7f49d62c9700 1 --2- 172.21.15.78:0/2393682697 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49d00a4ec0 0x7f49d00beca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.552+0000 7f49d62c9700 1 --2- 172.21.15.78:0/2393682697 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f49d00c3fd0 0x7f49d00893a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.552+0000 7f49d62c9700 1 -- 172.21.15.78:0/2393682697 >> 172.21.15.78:0/2393682697 conn(0x7f49d006aab0 msgr2=0x7f49d0164e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:23.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.552+0000 7f49d62c9700 1 -- 172.21.15.78:0/2393682697 shutdown_connections 2023-10-14T11:58:23.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.552+0000 7f49d62c9700 1 -- 172.21.15.78:0/2393682697 wait complete. 2023-10-14T11:58:23.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.554+0000 7f86d12f4700 1 -- 172.21.15.78:0/4142975443 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f86b806af30 msgr2=0x7f86b806d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.554+0000 7f86d12f4700 1 --2- 172.21.15.78:0/4142975443 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f86b806af30 0x7f86b806d3f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f86c0000c00 tx=0x7f86c000a000 comp rx=0 tx=0).stop 2023-10-14T11:58:23.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.554+0000 7f86d12f4700 1 -- 172.21.15.78:0/4142975443 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86b40067f0 msgr2=0x7f86cc149c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.554+0000 7f86d12f4700 1 --2- 172.21.15.78:0/4142975443 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86b40067f0 0x7f86cc149c90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f86cc071210 tx=0x7f86bc00b670 comp rx=0 tx=0).stop 2023-10-14T11:58:23.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.554+0000 7f86d12f4700 1 -- 172.21.15.78:0/4142975443 shutdown_connections 2023-10-14T11:58:23.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.554+0000 7f86d12f4700 1 --2- 172.21.15.78:0/4142975443 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86cc0bbda0 0x7f86cc0ba3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.554+0000 7f86d12f4700 1 --2- 172.21.15.78:0/4142975443 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f86b806af30 0x7f86b806d3f0 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f86c0000c00 tx=0x7f86c000a000 comp rx=0 tx=0).stop 2023-10-14T11:58:23.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.554+0000 7f86d12f4700 1 --2- 172.21.15.78:0/4142975443 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86b40067f0 0x7f86cc149c90 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.554+0000 7f86d12f4700 1 --2- 172.21.15.78:0/4142975443 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f86cc154320 0x7f86cc147ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.554+0000 7f86d12f4700 1 -- 172.21.15.78:0/4142975443 >> 172.21.15.78:0/4142975443 conn(0x7f86cc0a84a0 msgr2=0x7f86cc0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:23.561 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.7 2023-10-14T11:58:23.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.557+0000 7f86d12f4700 1 -- 172.21.15.78:0/4142975443 shutdown_connections 2023-10-14T11:58:23.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.557+0000 7f86d12f4700 1 -- 172.21.15.78:0/4142975443 wait complete. 2023-10-14T11:58:23.562 DEBUG:teuthology.parallel:result is None 2023-10-14T11:58:23.563 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.6 2023-10-14T11:58:23.565 DEBUG:teuthology.parallel:result is None 2023-10-14T11:58:23.566 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.5 2023-10-14T11:58:23.566 DEBUG:teuthology.parallel:result is None 2023-10-14T11:58:23.570 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574853 for osd.2 2023-10-14T11:58:23.570 DEBUG:teuthology.parallel:result is None 2023-10-14T11:58:23.575 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.4 2023-10-14T11:58:23.576 DEBUG:teuthology.parallel:result is None 2023-10-14T11:58:23.576 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.1 2023-10-14T11:58:23.577 DEBUG:teuthology.parallel:result is None 2023-10-14T11:58:23.577 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.3 2023-10-14T11:58:23.578 DEBUG:teuthology.parallel:result is None 2023-10-14T11:58:23.578 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-14T11:58:23.578 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T11:58:23.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.736+0000 7f5167e18700 1 Processor -- start 2023-10-14T11:58:23.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.736+0000 7f5167e18700 1 -- start start 2023-10-14T11:58:23.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.737+0000 7f5167e18700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f51600c44c0 0x7f51600c7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.737+0000 7f5167e18700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f51600c72a0 0x7f51600c86e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.737+0000 7f5167e18700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5160156b20 con 0x7f51600c8c20 2023-10-14T11:58:23.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.737+0000 7f5167e18700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5160155f10 con 0x7f51600c72a0 2023-10-14T11:58:23.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.737+0000 7f5167e18700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51601561f0 con 0x7f51600c44c0 2023-10-14T11:58:23.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.737+0000 7f51653b3700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f51600c72a0 0x7f51600c86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.737+0000 7f51663b5700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f51600c8c20 0x7f51600c2c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49794/0 (socket says 172.21.15.78:49794) 2023-10-14T11:58:23.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.737+0000 7f51663b5700 1 -- 172.21.15.78:0/826193277 learned_addr learned my addr 172.21.15.78:0/826193277 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:23.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.737+0000 7f5165bb4700 1 --2- 172.21.15.78:0/826193277 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f51600c44c0 0x7f51600c7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.737+0000 7f51653b3700 1 -- 172.21.15.78:0/826193277 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f51600c44c0 msgr2=0x7f51600c7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:23.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.737+0000 7f51653b3700 1 --2- 172.21.15.78:0/826193277 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f51600c44c0 0x7f51600c7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.737+0000 7f51653b3700 1 -- 172.21.15.78:0/826193277 >> v1:172.21.15.78:6789/0 conn(0x7f51600c8c20 legacy=0x7f51600c2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.737+0000 7f51653b3700 1 -- 172.21.15.78:0/826193277 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51601564d0 con 0x7f51600c72a0 2023-10-14T11:58:23.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.737+0000 7f51653b3700 1 --2- 172.21.15.78:0/826193277 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51600c72a0 0x7f51600c86e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5150009840 tx=0x7f5150009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fc3720951db8e22 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:23.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.737+0000 7f5164bb2700 1 -- 172.21.15.78:0/826193277 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5150011830 con 0x7f51600c72a0 2023-10-14T11:58:23.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.737+0000 7f5164bb2700 1 -- 172.21.15.78:0/826193277 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5150011990 con 0x7f51600c72a0 2023-10-14T11:58:23.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.737+0000 7f5164bb2700 1 -- 172.21.15.78:0/826193277 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5150011d20 con 0x7f51600c72a0 2023-10-14T11:58:23.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.738+0000 7f5167e18700 1 -- 172.21.15.78:0/826193277 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51600c72a0 msgr2=0x7f51600c86e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.738+0000 7f5167e18700 1 --2- 172.21.15.78:0/826193277 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51600c72a0 0x7f51600c86e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5150009840 tx=0x7f5150009b50 comp rx=0 tx=0).stop 2023-10-14T11:58:23.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.738+0000 7f5167e18700 1 -- 172.21.15.78:0/826193277 shutdown_connections 2023-10-14T11:58:23.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.738+0000 7f5167e18700 1 --2- 172.21.15.78:0/826193277 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f51600c44c0 0x7f51600c7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.738+0000 7f5167e18700 1 --2- 172.21.15.78:0/826193277 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51600c72a0 0x7f51600c86e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.738+0000 7f5167e18700 1 -- 172.21.15.78:0/826193277 >> 172.21.15.78:0/826193277 conn(0x7f51600737d0 msgr2=0x7f5160073be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:23.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.738+0000 7f5167e18700 1 -- 172.21.15.78:0/826193277 shutdown_connections 2023-10-14T11:58:23.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.738+0000 7f5167e18700 1 -- 172.21.15.78:0/826193277 wait complete. 2023-10-14T11:58:23.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.739+0000 7f5167e18700 1 Processor -- start 2023-10-14T11:58:23.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.739+0000 7f5167e18700 1 -- start start 2023-10-14T11:58:23.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.740+0000 7f5167e18700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f51600c44c0 0x7f51600ac950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.740+0000 7f5167e18700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51600c72a0 0x7f51600a61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.740+0000 7f5167e18700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f51600c8c20 0x7f51600a6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.740+0000 7f5167e18700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51600adf00 con 0x7f51600c72a0 2023-10-14T11:58:23.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.740+0000 7f5167e18700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51600c19a0 con 0x7f51600c8c20 2023-10-14T11:58:23.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.740+0000 7f5167e18700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51600ab400 con 0x7f51600c44c0 2023-10-14T11:58:23.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.740+0000 7f51653b3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51600c72a0 0x7f51600a61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.740+0000 7f51653b3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51600c72a0 0x7f51600a61f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44924/0 (socket says 172.21.15.78:44924) 2023-10-14T11:58:23.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.740+0000 7f51653b3700 1 -- 172.21.15.78:0/1469911384 learned_addr learned my addr 172.21.15.78:0/1469911384 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:23.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.740+0000 7f5165bb4700 1 --2- 172.21.15.78:0/1469911384 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f51600c44c0 0x7f51600ac950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.740+0000 7f51653b3700 1 -- 172.21.15.78:0/1469911384 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f51600c44c0 msgr2=0x7f51600ac950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.740+0000 7f51663b5700 1 --2- 172.21.15.78:0/1469911384 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f51600c8c20 0x7f51600a6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.740+0000 7f51653b3700 1 --2- 172.21.15.78:0/1469911384 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f51600c44c0 0x7f51600ac950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.740+0000 7f51653b3700 1 -- 172.21.15.78:0/1469911384 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f51600c8c20 msgr2=0x7f51600a6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.740+0000 7f51653b3700 1 --2- 172.21.15.78:0/1469911384 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f51600c8c20 0x7f51600a6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.740+0000 7f51653b3700 1 -- 172.21.15.78:0/1469911384 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51500094a0 con 0x7f51600c72a0 2023-10-14T11:58:23.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.740+0000 7f5165bb4700 1 --2- 172.21.15.78:0/1469911384 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f51600c44c0 0x7f51600ac950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:23.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.740+0000 7f51663b5700 1 --2- 172.21.15.78:0/1469911384 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f51600c8c20 0x7f51600a6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:23.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.740+0000 7f51653b3700 1 --2- 172.21.15.78:0/1469911384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51600c72a0 0x7f51600a61f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5150009b90 tx=0x7f5150004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:23.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.740+0000 7f51567fc700 1 -- 172.21.15.78:0/1469911384 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51500036a0 con 0x7f51600c72a0 2023-10-14T11:58:23.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.740+0000 7f51567fc700 1 -- 172.21.15.78:0/1469911384 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f515001f070 con 0x7f51600c72a0 2023-10-14T11:58:23.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.740+0000 7f5167e18700 1 -- 172.21.15.78:0/1469911384 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51601ea9e0 con 0x7f51600c72a0 2023-10-14T11:58:23.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.740+0000 7f51567fc700 1 -- 172.21.15.78:0/1469911384 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5150003a40 con 0x7f51600c72a0 2023-10-14T11:58:23.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.740+0000 7f5167e18700 1 -- 172.21.15.78:0/1469911384 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51600ace90 con 0x7f51600c72a0 2023-10-14T11:58:23.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.741+0000 7f5167e18700 1 -- 172.21.15.78:0/1469911384 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51600ab5b0 con 0x7f51600c72a0 2023-10-14T11:58:23.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.742+0000 7f51567fc700 1 -- 172.21.15.78:0/1469911384 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f5150003ba0 con 0x7f51600c72a0 2023-10-14T11:58:23.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.742+0000 7f51567fc700 1 --2- 172.21.15.78:0/1469911384 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f514406af00 0x7f514406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:23.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.742+0000 7f5165bb4700 1 --2- 172.21.15.78:0/1469911384 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f514406af00 0x7f514406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:23.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.742+0000 7f51567fc700 1 -- 172.21.15.78:0/1469911384 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5150021070 con 0x7f51600c72a0 2023-10-14T11:58:23.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.743+0000 7f5165bb4700 1 --2- 172.21.15.78:0/1469911384 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f514406af00 0x7f514406d3c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f515c009a00 tx=0x7f515c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:23.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.747+0000 7f51567fc700 1 -- 172.21.15.78:0/1469911384 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f51600ab5b0 con 0x7f51600c72a0 2023-10-14T11:58:23.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.935+0000 7f5167e18700 1 -- 172.21.15.78:0/1469911384 --> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f51600c48a0 con 0x7f514406af00 2023-10-14T11:58:23.944 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.942+0000 7f51567fc700 1 -- 172.21.15.78:0/1469911384 <== mgr.4104 v2:172.21.15.78:6832/3171890074 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264576 (secure 0 0 0) 0x7f515c009610 con 0x7f514406af00 2023-10-14T11:58:23.947 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:23.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.945+0000 7f5167e18700 1 -- 172.21.15.78:0/1469911384 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f514406af00 msgr2=0x7f514406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.945+0000 7f5167e18700 1 --2- 172.21.15.78:0/1469911384 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f514406af00 0x7f514406d3c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f515c009a00 tx=0x7f515c008040 comp rx=0 tx=0).stop 2023-10-14T11:58:23.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.945+0000 7f5167e18700 1 -- 172.21.15.78:0/1469911384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51600c72a0 msgr2=0x7f51600a61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:23.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.945+0000 7f5167e18700 1 --2- 172.21.15.78:0/1469911384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51600c72a0 0x7f51600a61f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5150009b90 tx=0x7f5150004750 comp rx=0 tx=0).stop 2023-10-14T11:58:23.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.945+0000 7f5167e18700 1 -- 172.21.15.78:0/1469911384 shutdown_connections 2023-10-14T11:58:23.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.945+0000 7f5167e18700 1 --2- 172.21.15.78:0/1469911384 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f514406af00 0x7f514406d3c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.945+0000 7f5167e18700 1 --2- 172.21.15.78:0/1469911384 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f51600c8c20 0x7f51600a6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.945+0000 7f5167e18700 1 --2- 172.21.15.78:0/1469911384 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f51600c44c0 0x7f51600ac950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.945+0000 7f5167e18700 1 --2- 172.21.15.78:0/1469911384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51600c72a0 0x7f51600a61f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:23.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.945+0000 7f5167e18700 1 -- 172.21.15.78:0/1469911384 >> 172.21.15.78:0/1469911384 conn(0x7f51600737d0 msgr2=0x7f51600c4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:23.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.945+0000 7f5167e18700 1 -- 172.21.15.78:0/1469911384 shutdown_connections 2023-10-14T11:58:23.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:23.945+0000 7f5167e18700 1 -- 172.21.15.78:0/1469911384 wait complete. 2023-10-14T11:58:23.952 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2023-10-14T11:58:23.959 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-14T11:58:22.966197+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218080,"kb_used_data":1888,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756640,"statfs":{"total":773094113280,"available":772870799360,"internally_reserved":0,"allocated":1933312,"data_stored":1790108,"data_compressed":65244,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.008217"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297796+0000","last_change":"2023-10-14T11:58:15.161449+0000","last_active":"2023-10-14T11:58:16.297796+0000","last_peered":"2023-10-14T11:58:16.297796+0000","last_clean":"2023-10-14T11:58:16.297796+0000","last_became_active":"2023-10-14T11:58:15.161100+0000","last_became_peered":"2023-10-14T11:58:15.161100+0000","last_unstale":"2023-10-14T11:58:16.297796+0000","last_undegraded":"2023-10-14T11:58:16.297796+0000","last_fullsized":"2023-10-14T11:58:16.297796+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:47:58.533238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.145107+0000","last_change":"2023-10-14T11:58:15.158210+0000","last_active":"2023-10-14T11:58:16.145107+0000","last_peered":"2023-10-14T11:58:16.145107+0000","last_clean":"2023-10-14T11:58:16.145107+0000","last_became_active":"2023-10-14T11:58:15.157886+0000","last_became_peered":"2023-10-14T11:58:15.157886+0000","last_unstale":"2023-10-14T11:58:16.145107+0000","last_undegraded":"2023-10-14T11:58:16.145107+0000","last_fullsized":"2023-10-14T11:58:16.145107+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:23:17.281352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296226+0000","last_change":"2023-10-14T11:58:15.161880+0000","last_active":"2023-10-14T11:58:16.296226+0000","last_peered":"2023-10-14T11:58:16.296226+0000","last_clean":"2023-10-14T11:58:16.296226+0000","last_became_active":"2023-10-14T11:58:15.161603+0000","last_became_peered":"2023-10-14T11:58:15.161603+0000","last_unstale":"2023-10-14T11:58:16.296226+0000","last_undegraded":"2023-10-14T11:58:16.296226+0000","last_fullsized":"2023-10-14T11:58:16.296226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:01:03.783116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296854+0000","last_change":"2023-10-14T11:58:15.167778+0000","last_active":"2023-10-14T11:58:16.296854+0000","last_peered":"2023-10-14T11:58:16.296854+0000","last_clean":"2023-10-14T11:58:16.296854+0000","last_became_active":"2023-10-14T11:58:15.167471+0000","last_became_peered":"2023-10-14T11:58:15.167471+0000","last_unstale":"2023-10-14T11:58:16.296854+0000","last_undegraded":"2023-10-14T11:58:16.296854+0000","last_fullsized":"2023-10-14T11:58:16.296854+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:07:58.884850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296653+0000","last_change":"2023-10-14T11:58:15.160788+0000","last_active":"2023-10-14T11:58:16.296653+0000","last_peered":"2023-10-14T11:58:16.296653+0000","last_clean":"2023-10-14T11:58:16.296653+0000","last_became_active":"2023-10-14T11:58:15.160525+0000","last_became_peered":"2023-10-14T11:58:15.160525+0000","last_unstale":"2023-10-14T11:58:16.296653+0000","last_undegraded":"2023-10-14T11:58:16.296653+0000","last_fullsized":"2023-10-14T11:58:16.296653+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:35:58.025887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297023+0000","last_change":"2023-10-14T11:58:15.167657+0000","last_active":"2023-10-14T11:58:16.297023+0000","last_peered":"2023-10-14T11:58:16.297023+0000","last_clean":"2023-10-14T11:58:16.297023+0000","last_became_active":"2023-10-14T11:58:15.167383+0000","last_became_peered":"2023-10-14T11:58:15.167383+0000","last_unstale":"2023-10-14T11:58:16.297023+0000","last_undegraded":"2023-10-14T11:58:16.297023+0000","last_fullsized":"2023-10-14T11:58:16.297023+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:53:19.926796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296834+0000","last_change":"2023-10-14T11:58:15.156244+0000","last_active":"2023-10-14T11:58:16.296834+0000","last_peered":"2023-10-14T11:58:16.296834+0000","last_clean":"2023-10-14T11:58:16.296834+0000","last_became_active":"2023-10-14T11:58:15.155621+0000","last_became_peered":"2023-10-14T11:58:15.155621+0000","last_unstale":"2023-10-14T11:58:16.296834+0000","last_undegraded":"2023-10-14T11:58:16.296834+0000","last_fullsized":"2023-10-14T11:58:16.296834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:07:58.884850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298542+0000","last_change":"2023-10-14T11:58:15.165865+0000","last_active":"2023-10-14T11:58:16.298542+0000","last_peered":"2023-10-14T11:58:16.298542+0000","last_clean":"2023-10-14T11:58:16.298542+0000","last_became_active":"2023-10-14T11:58:15.165613+0000","last_became_peered":"2023-10-14T11:58:15.165613+0000","last_unstale":"2023-10-14T11:58:16.298542+0000","last_undegraded":"2023-10-14T11:58:16.298542+0000","last_fullsized":"2023-10-14T11:58:16.298542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:32:38.145488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297728+0000","last_change":"2023-10-14T11:58:15.167047+0000","last_active":"2023-10-14T11:58:16.297728+0000","last_peered":"2023-10-14T11:58:16.297728+0000","last_clean":"2023-10-14T11:58:16.297728+0000","last_became_active":"2023-10-14T11:58:15.166362+0000","last_became_peered":"2023-10-14T11:58:15.166362+0000","last_unstale":"2023-10-14T11:58:16.297728+0000","last_undegraded":"2023-10-14T11:58:16.297728+0000","last_fullsized":"2023-10-14T11:58:16.297728+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:39:27.703491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144707+0000","last_change":"2023-10-14T11:58:15.164412+0000","last_active":"2023-10-14T11:58:16.144707+0000","last_peered":"2023-10-14T11:58:16.144707+0000","last_clean":"2023-10-14T11:58:16.144707+0000","last_became_active":"2023-10-14T11:58:15.164187+0000","last_became_peered":"2023-10-14T11:58:15.164187+0000","last_unstale":"2023-10-14T11:58:16.144707+0000","last_undegraded":"2023-10-14T11:58:16.144707+0000","last_fullsized":"2023-10-14T11:58:16.144707+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:59:03.464936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144466+0000","last_change":"2023-10-14T11:58:15.163857+0000","last_active":"2023-10-14T11:58:16.144466+0000","last_peered":"2023-10-14T11:58:16.144466+0000","last_clean":"2023-10-14T11:58:16.144466+0000","last_became_active":"2023-10-14T11:58:15.163575+0000","last_became_peered":"2023-10-14T11:58:15.163575+0000","last_unstale":"2023-10-14T11:58:16.144466+0000","last_undegraded":"2023-10-14T11:58:16.144466+0000","last_fullsized":"2023-10-14T11:58:16.144466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:01:03.783116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144222+0000","last_change":"2023-10-14T11:58:15.149569+0000","last_active":"2023-10-14T11:58:16.144222+0000","last_peered":"2023-10-14T11:58:16.144222+0000","last_clean":"2023-10-14T11:58:16.144222+0000","last_became_active":"2023-10-14T11:58:15.149161+0000","last_became_peered":"2023-10-14T11:58:15.149161+0000","last_unstale":"2023-10-14T11:58:16.144222+0000","last_undegraded":"2023-10-14T11:58:16.144222+0000","last_fullsized":"2023-10-14T11:58:16.144222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:19:30.143095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.299094+0000","last_change":"2023-10-14T11:58:15.147173+0000","last_active":"2023-10-14T11:58:16.299094+0000","last_peered":"2023-10-14T11:58:16.299094+0000","last_clean":"2023-10-14T11:58:16.299094+0000","last_became_active":"2023-10-14T11:58:15.146910+0000","last_became_peered":"2023-10-14T11:58:15.146910+0000","last_unstale":"2023-10-14T11:58:16.299094+0000","last_undegraded":"2023-10-14T11:58:16.299094+0000","last_fullsized":"2023-10-14T11:58:16.299094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:59:03.464936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296040+0000","last_change":"2023-10-14T11:58:15.169906+0000","last_active":"2023-10-14T11:58:16.296040+0000","last_peered":"2023-10-14T11:58:16.296040+0000","last_clean":"2023-10-14T11:58:16.296040+0000","last_became_active":"2023-10-14T11:58:15.169660+0000","last_became_peered":"2023-10-14T11:58:15.169660+0000","last_unstale":"2023-10-14T11:58:16.296040+0000","last_undegraded":"2023-10-14T11:58:16.296040+0000","last_fullsized":"2023-10-14T11:58:16.296040+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:32:38.145488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144820+0000","last_change":"2023-10-14T11:58:15.159353+0000","last_active":"2023-10-14T11:58:16.144820+0000","last_peered":"2023-10-14T11:58:16.144820+0000","last_clean":"2023-10-14T11:58:16.144820+0000","last_became_active":"2023-10-14T11:58:15.159095+0000","last_became_peered":"2023-10-14T11:58:15.159095+0000","last_unstale":"2023-10-14T11:58:16.144820+0000","last_undegraded":"2023-10-14T11:58:16.144820+0000","last_fullsized":"2023-10-14T11:58:16.144820+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:33:00.861183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296349+0000","last_change":"2023-10-14T11:58:15.169061+0000","last_active":"2023-10-14T11:58:16.296349+0000","last_peered":"2023-10-14T11:58:16.296349+0000","last_clean":"2023-10-14T11:58:16.296349+0000","last_became_active":"2023-10-14T11:58:15.168627+0000","last_became_peered":"2023-10-14T11:58:15.168627+0000","last_unstale":"2023-10-14T11:58:16.296349+0000","last_undegraded":"2023-10-14T11:58:16.296349+0000","last_fullsized":"2023-10-14T11:58:16.296349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:23:17.281352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297017+0000","last_change":"2023-10-14T11:58:15.162773+0000","last_active":"2023-10-14T11:58:16.297017+0000","last_peered":"2023-10-14T11:58:16.297017+0000","last_clean":"2023-10-14T11:58:16.297017+0000","last_became_active":"2023-10-14T11:58:15.162514+0000","last_became_peered":"2023-10-14T11:58:15.162514+0000","last_unstale":"2023-10-14T11:58:16.297017+0000","last_undegraded":"2023-10-14T11:58:16.297017+0000","last_fullsized":"2023-10-14T11:58:16.297017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:02:52.001714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298805+0000","last_change":"2023-10-14T11:58:15.164541+0000","last_active":"2023-10-14T11:58:16.298805+0000","last_peered":"2023-10-14T11:58:16.298805+0000","last_clean":"2023-10-14T11:58:16.298805+0000","last_became_active":"2023-10-14T11:58:15.164309+0000","last_became_peered":"2023-10-14T11:58:15.164309+0000","last_unstale":"2023-10-14T11:58:16.298805+0000","last_undegraded":"2023-10-14T11:58:16.298805+0000","last_fullsized":"2023-10-14T11:58:16.298805+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:22:04.803877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296451+0000","last_change":"2023-10-14T11:58:15.162171+0000","last_active":"2023-10-14T11:58:16.296451+0000","last_peered":"2023-10-14T11:58:16.296451+0000","last_clean":"2023-10-14T11:58:16.296451+0000","last_became_active":"2023-10-14T11:58:15.161894+0000","last_became_peered":"2023-10-14T11:58:15.161894+0000","last_unstale":"2023-10-14T11:58:16.296451+0000","last_undegraded":"2023-10-14T11:58:16.296451+0000","last_fullsized":"2023-10-14T11:58:16.296451+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:15:07.231894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296298+0000","last_change":"2023-10-14T11:58:14.145266+0000","last_active":"2023-10-14T11:58:16.296298+0000","last_peered":"2023-10-14T11:58:16.296298+0000","last_clean":"2023-10-14T11:58:16.296298+0000","last_became_active":"2023-10-14T11:58:14.144885+0000","last_became_peered":"2023-10-14T11:58:14.144885+0000","last_unstale":"2023-10-14T11:58:16.296298+0000","last_undegraded":"2023-10-14T11:58:16.296298+0000","last_fullsized":"2023-10-14T11:58:16.296298+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:06:27.985367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.299066+0000","last_change":"2023-10-14T11:58:15.148017+0000","last_active":"2023-10-14T11:58:16.299066+0000","last_peered":"2023-10-14T11:58:16.299066+0000","last_clean":"2023-10-14T11:58:16.299066+0000","last_became_active":"2023-10-14T11:58:15.147796+0000","last_became_peered":"2023-10-14T11:58:15.147796+0000","last_unstale":"2023-10-14T11:58:16.299066+0000","last_undegraded":"2023-10-14T11:58:16.299066+0000","last_fullsized":"2023-10-14T11:58:16.299066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:23:17.281352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.299129+0000","last_change":"2023-10-14T11:58:14.152335+0000","last_active":"2023-10-14T11:58:16.299129+0000","last_peered":"2023-10-14T11:58:16.299129+0000","last_clean":"2023-10-14T11:58:16.299129+0000","last_became_active":"2023-10-14T11:58:14.152064+0000","last_became_peered":"2023-10-14T11:58:14.152064+0000","last_unstale":"2023-10-14T11:58:16.299129+0000","last_undegraded":"2023-10-14T11:58:16.299129+0000","last_fullsized":"2023-10-14T11:58:16.299129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:21:22.993387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296225+0000","last_change":"2023-10-14T11:58:15.169510+0000","last_active":"2023-10-14T11:58:16.296225+0000","last_peered":"2023-10-14T11:58:16.296225+0000","last_clean":"2023-10-14T11:58:16.296225+0000","last_became_active":"2023-10-14T11:58:15.166488+0000","last_became_peered":"2023-10-14T11:58:15.166488+0000","last_unstale":"2023-10-14T11:58:16.296225+0000","last_undegraded":"2023-10-14T11:58:16.296225+0000","last_fullsized":"2023-10-14T11:58:16.296225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:23:17.281352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297447+0000","last_change":"2023-10-14T11:58:14.150333+0000","last_active":"2023-10-14T11:58:16.297447+0000","last_peered":"2023-10-14T11:58:16.297447+0000","last_clean":"2023-10-14T11:58:16.297447+0000","last_became_active":"2023-10-14T11:58:14.149979+0000","last_became_peered":"2023-10-14T11:58:14.149979+0000","last_unstale":"2023-10-14T11:58:16.297447+0000","last_undegraded":"2023-10-14T11:58:16.297447+0000","last_fullsized":"2023-10-14T11:58:16.297447+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:23:16.277717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297176+0000","last_change":"2023-10-14T11:58:15.163307+0000","last_active":"2023-10-14T11:58:16.297176+0000","last_peered":"2023-10-14T11:58:16.297176+0000","last_clean":"2023-10-14T11:58:16.297176+0000","last_became_active":"2023-10-14T11:58:15.163015+0000","last_became_peered":"2023-10-14T11:58:15.163015+0000","last_unstale":"2023-10-14T11:58:16.297176+0000","last_undegraded":"2023-10-14T11:58:16.297176+0000","last_fullsized":"2023-10-14T11:58:16.297176+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:21:10.166149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296924+0000","last_change":"2023-10-14T11:58:14.150931+0000","last_active":"2023-10-14T11:58:16.296924+0000","last_peered":"2023-10-14T11:58:16.296924+0000","last_clean":"2023-10-14T11:58:16.296924+0000","last_became_active":"2023-10-14T11:58:14.150629+0000","last_became_peered":"2023-10-14T11:58:14.150629+0000","last_unstale":"2023-10-14T11:58:16.296924+0000","last_undegraded":"2023-10-14T11:58:16.296924+0000","last_fullsized":"2023-10-14T11:58:16.296924+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:22:03.800242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297270+0000","last_change":"2023-10-14T11:58:15.163629+0000","last_active":"2023-10-14T11:58:16.297270+0000","last_peered":"2023-10-14T11:58:16.297270+0000","last_clean":"2023-10-14T11:58:16.297270+0000","last_became_active":"2023-10-14T11:58:15.163043+0000","last_became_peered":"2023-10-14T11:58:15.163043+0000","last_unstale":"2023-10-14T11:58:16.297270+0000","last_undegraded":"2023-10-14T11:58:16.297270+0000","last_fullsized":"2023-10-14T11:58:16.297270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:39:27.703491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298703+0000","last_change":"2023-10-14T11:58:14.154507+0000","last_active":"2023-10-14T11:58:16.298703+0000","last_peered":"2023-10-14T11:58:16.298703+0000","last_clean":"2023-10-14T11:58:16.298703+0000","last_became_active":"2023-10-14T11:58:14.154202+0000","last_became_peered":"2023-10-14T11:58:14.154202+0000","last_unstale":"2023-10-14T11:58:16.298703+0000","last_undegraded":"2023-10-14T11:58:16.298703+0000","last_fullsized":"2023-10-14T11:58:16.298703+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:59:45.791176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.143513+0000","last_change":"2023-10-14T11:58:15.158639+0000","last_active":"2023-10-14T11:58:16.143513+0000","last_peered":"2023-10-14T11:58:16.143513+0000","last_clean":"2023-10-14T11:58:16.143513+0000","last_became_active":"2023-10-14T11:58:15.158402+0000","last_became_peered":"2023-10-14T11:58:15.158402+0000","last_unstale":"2023-10-14T11:58:16.143513+0000","last_undegraded":"2023-10-14T11:58:16.143513+0000","last_fullsized":"2023-10-14T11:58:16.143513+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:21:23.997022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144263+0000","last_change":"2023-10-14T11:58:14.139502+0000","last_active":"2023-10-14T11:58:16.144263+0000","last_peered":"2023-10-14T11:58:16.144263+0000","last_clean":"2023-10-14T11:58:16.144263+0000","last_became_active":"2023-10-14T11:58:14.139053+0000","last_became_peered":"2023-10-14T11:58:14.139053+0000","last_unstale":"2023-10-14T11:58:16.144263+0000","last_undegraded":"2023-10-14T11:58:16.144263+0000","last_fullsized":"2023-10-14T11:58:16.144263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:10:10.291507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296162+0000","last_change":"2023-10-14T11:58:15.169858+0000","last_active":"2023-10-14T11:58:16.296162+0000","last_peered":"2023-10-14T11:58:16.296162+0000","last_clean":"2023-10-14T11:58:16.296162+0000","last_became_active":"2023-10-14T11:58:15.166724+0000","last_became_peered":"2023-10-14T11:58:15.166724+0000","last_unstale":"2023-10-14T11:58:16.296162+0000","last_undegraded":"2023-10-14T11:58:16.296162+0000","last_fullsized":"2023-10-14T11:58:16.296162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:45:35.707554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.299064+0000","last_change":"2023-10-14T11:58:14.154253+0000","last_active":"2023-10-14T11:58:16.299064+0000","last_peered":"2023-10-14T11:58:16.299064+0000","last_clean":"2023-10-14T11:58:16.299064+0000","last_became_active":"2023-10-14T11:58:14.152400+0000","last_became_peered":"2023-10-14T11:58:14.152400+0000","last_unstale":"2023-10-14T11:58:16.299064+0000","last_undegraded":"2023-10-14T11:58:16.299064+0000","last_fullsized":"2023-10-14T11:58:16.299064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:03:15.889637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298935+0000","last_change":"2023-10-14T11:58:15.167532+0000","last_active":"2023-10-14T11:58:16.298935+0000","last_peered":"2023-10-14T11:58:16.298935+0000","last_clean":"2023-10-14T11:58:16.298935+0000","last_became_active":"2023-10-14T11:58:15.167270+0000","last_became_peered":"2023-10-14T11:58:15.167270+0000","last_unstale":"2023-10-14T11:58:16.298935+0000","last_undegraded":"2023-10-14T11:58:16.298935+0000","last_fullsized":"2023-10-14T11:58:16.298935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:33:00.861183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296354+0000","last_change":"2023-10-14T11:58:14.147092+0000","last_active":"2023-10-14T11:58:16.296354+0000","last_peered":"2023-10-14T11:58:16.296354+0000","last_clean":"2023-10-14T11:58:16.296354+0000","last_became_active":"2023-10-14T11:58:14.146377+0000","last_became_peered":"2023-10-14T11:58:14.146377+0000","last_unstale":"2023-10-14T11:58:16.296354+0000","last_undegraded":"2023-10-14T11:58:16.296354+0000","last_fullsized":"2023-10-14T11:58:16.296354+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:25:37.999389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144924+0000","last_change":"2023-10-14T11:58:15.160286+0000","last_active":"2023-10-14T11:58:16.144924+0000","last_peered":"2023-10-14T11:58:16.144924+0000","last_clean":"2023-10-14T11:58:16.144924+0000","last_became_active":"2023-10-14T11:58:15.159308+0000","last_became_peered":"2023-10-14T11:58:15.159308+0000","last_unstale":"2023-10-14T11:58:16.144924+0000","last_undegraded":"2023-10-14T11:58:16.144924+0000","last_fullsized":"2023-10-14T11:58:16.144924+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:45:35.707554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144601+0000","last_change":"2023-10-14T11:58:14.148448+0000","last_active":"2023-10-14T11:58:16.144601+0000","last_peered":"2023-10-14T11:58:16.144601+0000","last_clean":"2023-10-14T11:58:16.144601+0000","last_became_active":"2023-10-14T11:58:14.148155+0000","last_became_peered":"2023-10-14T11:58:14.148155+0000","last_unstale":"2023-10-14T11:58:16.144601+0000","last_undegraded":"2023-10-14T11:58:16.144601+0000","last_fullsized":"2023-10-14T11:58:16.144601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:19:29.139460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296785+0000","last_change":"2023-10-14T11:58:15.166089+0000","last_active":"2023-10-14T11:58:16.296785+0000","last_peered":"2023-10-14T11:58:16.296785+0000","last_clean":"2023-10-14T11:58:16.296785+0000","last_became_active":"2023-10-14T11:58:15.164889+0000","last_became_peered":"2023-10-14T11:58:15.164889+0000","last_unstale":"2023-10-14T11:58:16.296785+0000","last_undegraded":"2023-10-14T11:58:16.296785+0000","last_fullsized":"2023-10-14T11:58:16.296785+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:15.482221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296897+0000","last_change":"2023-10-14T11:58:14.150455+0000","last_active":"2023-10-14T11:58:16.296897+0000","last_peered":"2023-10-14T11:58:16.296897+0000","last_clean":"2023-10-14T11:58:16.296897+0000","last_became_active":"2023-10-14T11:58:14.150195+0000","last_became_peered":"2023-10-14T11:58:14.150195+0000","last_unstale":"2023-10-14T11:58:16.296897+0000","last_undegraded":"2023-10-14T11:58:16.296897+0000","last_fullsized":"2023-10-14T11:58:16.296897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:32:59.857548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296229+0000","last_change":"2023-10-14T11:58:15.170459+0000","last_active":"2023-10-14T11:58:16.296229+0000","last_peered":"2023-10-14T11:58:16.296229+0000","last_clean":"2023-10-14T11:58:16.296229+0000","last_became_active":"2023-10-14T11:58:15.170096+0000","last_became_peered":"2023-10-14T11:58:15.170096+0000","last_unstale":"2023-10-14T11:58:16.296229+0000","last_undegraded":"2023-10-14T11:58:16.296229+0000","last_fullsized":"2023-10-14T11:58:16.296229+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:21:15.443882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.145168+0000","last_change":"2023-10-14T11:58:14.147259+0000","last_active":"2023-10-14T11:58:16.145168+0000","last_peered":"2023-10-14T11:58:16.145168+0000","last_clean":"2023-10-14T11:58:16.145168+0000","last_became_active":"2023-10-14T11:58:14.146982+0000","last_became_peered":"2023-10-14T11:58:14.146982+0000","last_unstale":"2023-10-14T11:58:16.145168+0000","last_undegraded":"2023-10-14T11:58:16.145168+0000","last_fullsized":"2023-10-14T11:58:16.145168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:22:03.800242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144175+0000","last_change":"2023-10-14T11:58:15.158917+0000","last_active":"2023-10-14T11:58:16.144175+0000","last_peered":"2023-10-14T11:58:16.144175+0000","last_clean":"2023-10-14T11:58:16.144175+0000","last_became_active":"2023-10-14T11:58:15.158578+0000","last_became_peered":"2023-10-14T11:58:15.158578+0000","last_unstale":"2023-10-14T11:58:16.144175+0000","last_undegraded":"2023-10-14T11:58:16.144175+0000","last_fullsized":"2023-10-14T11:58:16.144175+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:06:28.989002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144380+0000","last_change":"2023-10-14T11:58:14.152849+0000","last_active":"2023-10-14T11:58:16.144380+0000","last_peered":"2023-10-14T11:58:16.144380+0000","last_clean":"2023-10-14T11:58:16.144380+0000","last_became_active":"2023-10-14T11:58:14.152544+0000","last_became_peered":"2023-10-14T11:58:14.152544+0000","last_unstale":"2023-10-14T11:58:16.144380+0000","last_undegraded":"2023-10-14T11:58:16.144380+0000","last_fullsized":"2023-10-14T11:58:16.144380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:30:50.303297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297109+0000","last_change":"2023-10-14T11:58:15.160605+0000","last_active":"2023-10-14T11:58:16.297109+0000","last_peered":"2023-10-14T11:58:16.297109+0000","last_clean":"2023-10-14T11:58:16.297109+0000","last_became_active":"2023-10-14T11:58:15.160323+0000","last_became_peered":"2023-10-14T11:58:15.160323+0000","last_unstale":"2023-10-14T11:58:16.297109+0000","last_undegraded":"2023-10-14T11:58:16.297109+0000","last_fullsized":"2023-10-14T11:58:16.297109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:53:19.926796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296162+0000","last_change":"2023-10-14T11:58:14.145911+0000","last_active":"2023-10-14T11:58:16.296162+0000","last_peered":"2023-10-14T11:58:16.296162+0000","last_clean":"2023-10-14T11:58:16.296162+0000","last_became_active":"2023-10-14T11:58:14.144322+0000","last_became_peered":"2023-10-14T11:58:14.144322+0000","last_unstale":"2023-10-14T11:58:16.296162+0000","last_undegraded":"2023-10-14T11:58:16.296162+0000","last_fullsized":"2023-10-14T11:58:16.296162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:32:03.667630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.295965+0000","last_change":"2023-10-14T11:58:15.165189+0000","last_active":"2023-10-14T11:58:16.295965+0000","last_peered":"2023-10-14T11:58:16.295965+0000","last_clean":"2023-10-14T11:58:16.295965+0000","last_became_active":"2023-10-14T11:58:15.164911+0000","last_became_peered":"2023-10-14T11:58:15.164911+0000","last_unstale":"2023-10-14T11:58:16.295965+0000","last_undegraded":"2023-10-14T11:58:16.295965+0000","last_fullsized":"2023-10-14T11:58:16.295965+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:10:17.502466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297270+0000","last_change":"2023-10-14T11:58:14.152736+0000","last_active":"2023-10-14T11:58:16.297270+0000","last_peered":"2023-10-14T11:58:16.297270+0000","last_clean":"2023-10-14T11:58:16.297270+0000","last_became_active":"2023-10-14T11:58:14.152401+0000","last_became_peered":"2023-10-14T11:58:14.152401+0000","last_unstale":"2023-10-14T11:58:16.297270+0000","last_undegraded":"2023-10-14T11:58:16.297270+0000","last_fullsized":"2023-10-14T11:58:16.297270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:01:02.779481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296948+0000","last_change":"2023-10-14T11:58:15.155365+0000","last_active":"2023-10-14T11:58:16.296948+0000","last_peered":"2023-10-14T11:58:16.296948+0000","last_clean":"2023-10-14T11:58:16.296948+0000","last_became_active":"2023-10-14T11:58:15.155102+0000","last_became_peered":"2023-10-14T11:58:15.155102+0000","last_unstale":"2023-10-14T11:58:16.296948+0000","last_undegraded":"2023-10-14T11:58:16.296948+0000","last_fullsized":"2023-10-14T11:58:16.296948+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:02:52.001714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.143938+0000","last_change":"2023-10-14T11:58:14.148862+0000","last_active":"2023-10-14T11:58:16.143938+0000","last_peered":"2023-10-14T11:58:16.143938+0000","last_clean":"2023-10-14T11:58:16.143938+0000","last_became_active":"2023-10-14T11:58:14.148409+0000","last_became_peered":"2023-10-14T11:58:14.148409+0000","last_unstale":"2023-10-14T11:58:16.143938+0000","last_undegraded":"2023-10-14T11:58:16.143938+0000","last_fullsized":"2023-10-14T11:58:16.143938+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:06:27.985367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.143490+0000","last_change":"2023-10-14T11:58:15.156428+0000","last_active":"2023-10-14T11:58:16.143490+0000","last_peered":"2023-10-14T11:58:16.143490+0000","last_clean":"2023-10-14T11:58:16.143490+0000","last_became_active":"2023-10-14T11:58:15.155818+0000","last_became_peered":"2023-10-14T11:58:15.155818+0000","last_unstale":"2023-10-14T11:58:16.143490+0000","last_undegraded":"2023-10-14T11:58:16.143490+0000","last_fullsized":"2023-10-14T11:58:16.143490+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:33:00.861183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298391+0000","last_change":"2023-10-14T11:58:14.149643+0000","last_active":"2023-10-14T11:58:16.298391+0000","last_peered":"2023-10-14T11:58:16.298391+0000","last_clean":"2023-10-14T11:58:16.298391+0000","last_became_active":"2023-10-14T11:58:14.149330+0000","last_became_peered":"2023-10-14T11:58:14.149330+0000","last_unstale":"2023-10-14T11:58:16.298391+0000","last_undegraded":"2023-10-14T11:58:16.298391+0000","last_fullsized":"2023-10-14T11:58:16.298391+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:14:56.067093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296718+0000","last_change":"2023-10-14T11:58:15.156412+0000","last_active":"2023-10-14T11:58:16.296718+0000","last_peered":"2023-10-14T11:58:16.296718+0000","last_clean":"2023-10-14T11:58:16.296718+0000","last_became_active":"2023-10-14T11:58:15.155951+0000","last_became_peered":"2023-10-14T11:58:15.155951+0000","last_unstale":"2023-10-14T11:58:16.296718+0000","last_undegraded":"2023-10-14T11:58:16.296718+0000","last_fullsized":"2023-10-14T11:58:16.296718+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:15.482221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.143832+0000","last_change":"2023-10-14T11:58:14.135648+0000","last_active":"2023-10-14T11:58:16.143832+0000","last_peered":"2023-10-14T11:58:16.143832+0000","last_clean":"2023-10-14T11:58:16.143832+0000","last_became_active":"2023-10-14T11:58:14.135104+0000","last_became_peered":"2023-10-14T11:58:14.135104+0000","last_unstale":"2023-10-14T11:58:16.143832+0000","last_undegraded":"2023-10-14T11:58:16.143832+0000","last_fullsized":"2023-10-14T11:58:16.143832+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:47:41.184305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297156+0000","last_change":"2023-10-14T11:58:15.157354+0000","last_active":"2023-10-14T11:58:16.297156+0000","last_peered":"2023-10-14T11:58:16.297156+0000","last_clean":"2023-10-14T11:58:16.297156+0000","last_became_active":"2023-10-14T11:58:15.157084+0000","last_became_peered":"2023-10-14T11:58:15.157084+0000","last_unstale":"2023-10-14T11:58:16.297156+0000","last_undegraded":"2023-10-14T11:58:16.297156+0000","last_fullsized":"2023-10-14T11:58:16.297156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:32:40.379634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144935+0000","last_change":"2023-10-14T11:58:14.148246+0000","last_active":"2023-10-14T11:58:16.144935+0000","last_peered":"2023-10-14T11:58:16.144935+0000","last_clean":"2023-10-14T11:58:16.144935+0000","last_became_active":"2023-10-14T11:58:14.147943+0000","last_became_peered":"2023-10-14T11:58:14.147943+0000","last_unstale":"2023-10-14T11:58:16.144935+0000","last_undegraded":"2023-10-14T11:58:16.144935+0000","last_fullsized":"2023-10-14T11:58:16.144935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:19:29.139460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297125+0000","last_change":"2023-10-14T11:58:15.162066+0000","last_active":"2023-10-14T11:58:16.297125+0000","last_peered":"2023-10-14T11:58:16.297125+0000","last_clean":"2023-10-14T11:58:16.297125+0000","last_became_active":"2023-10-14T11:58:15.161798+0000","last_became_peered":"2023-10-14T11:58:15.161798+0000","last_unstale":"2023-10-14T11:58:16.297125+0000","last_undegraded":"2023-10-14T11:58:16.297125+0000","last_fullsized":"2023-10-14T11:58:16.297125+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:32:40.379634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297269+0000","last_change":"2023-10-14T11:58:14.147024+0000","last_active":"2023-10-14T11:58:16.297269+0000","last_peered":"2023-10-14T11:58:16.297269+0000","last_clean":"2023-10-14T11:58:16.297269+0000","last_became_active":"2023-10-14T11:58:14.146675+0000","last_became_peered":"2023-10-14T11:58:14.146675+0000","last_unstale":"2023-10-14T11:58:16.297269+0000","last_undegraded":"2023-10-14T11:58:16.297269+0000","last_fullsized":"2023-10-14T11:58:16.297269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:45:34.703919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.143832+0000","last_change":"2023-10-14T11:58:15.157978+0000","last_active":"2023-10-14T11:58:16.143832+0000","last_peered":"2023-10-14T11:58:16.143832+0000","last_clean":"2023-10-14T11:58:16.143832+0000","last_became_active":"2023-10-14T11:58:15.157698+0000","last_became_peered":"2023-10-14T11:58:15.157698+0000","last_unstale":"2023-10-14T11:58:16.143832+0000","last_undegraded":"2023-10-14T11:58:16.143832+0000","last_fullsized":"2023-10-14T11:58:16.143832+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:10:11.295142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144794+0000","last_change":"2023-10-14T11:58:15.164985+0000","last_active":"2023-10-14T11:58:16.144794+0000","last_peered":"2023-10-14T11:58:16.144794+0000","last_clean":"2023-10-14T11:58:16.144794+0000","last_became_active":"2023-10-14T11:58:15.164355+0000","last_became_peered":"2023-10-14T11:58:15.164355+0000","last_unstale":"2023-10-14T11:58:16.144794+0000","last_undegraded":"2023-10-14T11:58:16.144794+0000","last_fullsized":"2023-10-14T11:58:16.144794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:25:39.003024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298382+0000","last_change":"2023-10-14T11:58:14.153483+0000","last_active":"2023-10-14T11:58:16.298382+0000","last_peered":"2023-10-14T11:58:16.298382+0000","last_clean":"2023-10-14T11:58:16.298382+0000","last_became_active":"2023-10-14T11:58:14.153174+0000","last_became_peered":"2023-10-14T11:58:14.153174+0000","last_unstale":"2023-10-14T11:58:16.298382+0000","last_undegraded":"2023-10-14T11:58:16.298382+0000","last_fullsized":"2023-10-14T11:58:16.298382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:06:27.985367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298697+0000","last_change":"2023-10-14T11:58:15.171141+0000","last_active":"2023-10-14T11:58:16.298697+0000","last_peered":"2023-10-14T11:58:16.298697+0000","last_clean":"2023-10-14T11:58:16.298697+0000","last_became_active":"2023-10-14T11:58:15.170853+0000","last_became_peered":"2023-10-14T11:58:15.170853+0000","last_unstale":"2023-10-14T11:58:16.298697+0000","last_undegraded":"2023-10-14T11:58:16.298697+0000","last_fullsized":"2023-10-14T11:58:16.298697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:45:35.707554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296962+0000","last_change":"2023-10-14T11:58:14.153660+0000","last_active":"2023-10-14T11:58:16.296962+0000","last_peered":"2023-10-14T11:58:16.296962+0000","last_clean":"2023-10-14T11:58:16.296962+0000","last_became_active":"2023-10-14T11:58:14.153419+0000","last_became_peered":"2023-10-14T11:58:14.153419+0000","last_unstale":"2023-10-14T11:58:16.296962+0000","last_undegraded":"2023-10-14T11:58:16.296962+0000","last_fullsized":"2023-10-14T11:58:16.296962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:06:27.985367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T11:58:15.149401+0000","last_change":"2023-10-14T11:58:10.616726+0000","last_active":"2023-10-14T11:58:15.149401+0000","last_peered":"2023-10-14T11:58:15.149401+0000","last_clean":"2023-10-14T11:58:15.149401+0000","last_became_active":"2023-10-14T11:58:10.615628+0000","last_became_peered":"2023-10-14T11:58:10.615628+0000","last_unstale":"2023-10-14T11:58:15.149401+0000","last_undegraded":"2023-10-14T11:58:15.149401+0000","last_fullsized":"2023-10-14T11:58:15.149401+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:09.001265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:09.001265+0000","last_clean_scrub_stamp":"2023-10-14T11:58:09.001265+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:22:50.496050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.295927+0000","last_change":"2023-10-14T11:58:15.173632+0000","last_active":"2023-10-14T11:58:16.295927+0000","last_peered":"2023-10-14T11:58:16.295927+0000","last_clean":"2023-10-14T11:58:16.295927+0000","last_became_active":"2023-10-14T11:58:15.172996+0000","last_became_peered":"2023-10-14T11:58:15.172996+0000","last_unstale":"2023-10-14T11:58:16.295927+0000","last_undegraded":"2023-10-14T11:58:16.295927+0000","last_fullsized":"2023-10-14T11:58:16.295927+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:22:04.803877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296824+0000","last_change":"2023-10-14T11:58:14.150078+0000","last_active":"2023-10-14T11:58:16.296824+0000","last_peered":"2023-10-14T11:58:16.296824+0000","last_clean":"2023-10-14T11:58:16.296824+0000","last_became_active":"2023-10-14T11:58:14.149788+0000","last_became_peered":"2023-10-14T11:58:14.149788+0000","last_unstale":"2023-10-14T11:58:16.296824+0000","last_undegraded":"2023-10-14T11:58:16.296824+0000","last_fullsized":"2023-10-14T11:58:16.296824+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:10:10.291507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297353+0000","last_change":"2023-10-14T11:58:14.151386+0000","last_active":"2023-10-14T11:58:16.297353+0000","last_peered":"2023-10-14T11:58:16.297353+0000","last_clean":"2023-10-14T11:58:16.297353+0000","last_became_active":"2023-10-14T11:58:14.151041+0000","last_became_peered":"2023-10-14T11:58:14.151041+0000","last_unstale":"2023-10-14T11:58:16.297353+0000","last_undegraded":"2023-10-14T11:58:16.297353+0000","last_fullsized":"2023-10-14T11:58:16.297353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:21:14.440247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144942+0000","last_change":"2023-10-14T11:58:15.168958+0000","last_active":"2023-10-14T11:58:16.144942+0000","last_peered":"2023-10-14T11:58:16.144942+0000","last_clean":"2023-10-14T11:58:16.144942+0000","last_became_active":"2023-10-14T11:58:15.164516+0000","last_became_peered":"2023-10-14T11:58:15.164516+0000","last_unstale":"2023-10-14T11:58:16.144942+0000","last_undegraded":"2023-10-14T11:58:16.144942+0000","last_fullsized":"2023-10-14T11:58:16.144942+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:01:03.783116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:17.142950+0000","last_change":"2023-10-14T11:58:14.148617+0000","last_active":"2023-10-14T11:58:17.142950+0000","last_peered":"2023-10-14T11:58:17.142950+0000","last_clean":"2023-10-14T11:58:17.142950+0000","last_became_active":"2023-10-14T11:58:14.148324+0000","last_became_peered":"2023-10-14T11:58:14.148324+0000","last_unstale":"2023-10-14T11:58:17.142950+0000","last_undegraded":"2023-10-14T11:58:17.142950+0000","last_fullsized":"2023-10-14T11:58:17.142950+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:10:16.498831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.299219+0000","last_change":"2023-10-14T11:58:15.162441+0000","last_active":"2023-10-14T11:58:16.299219+0000","last_peered":"2023-10-14T11:58:16.299219+0000","last_clean":"2023-10-14T11:58:16.299219+0000","last_became_active":"2023-10-14T11:58:15.162047+0000","last_became_peered":"2023-10-14T11:58:15.162047+0000","last_unstale":"2023-10-14T11:58:16.299219+0000","last_undegraded":"2023-10-14T11:58:16.299219+0000","last_fullsized":"2023-10-14T11:58:16.299219+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:15.482221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144522+0000","last_change":"2023-10-14T11:58:14.151993+0000","last_active":"2023-10-14T11:58:16.144522+0000","last_peered":"2023-10-14T11:58:16.144522+0000","last_clean":"2023-10-14T11:58:16.144522+0000","last_became_active":"2023-10-14T11:58:14.151721+0000","last_became_peered":"2023-10-14T11:58:14.151721+0000","last_unstale":"2023-10-14T11:58:16.144522+0000","last_undegraded":"2023-10-14T11:58:16.144522+0000","last_fullsized":"2023-10-14T11:58:16.144522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:21:22.993387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296156+0000","last_change":"2023-10-14T11:58:15.170037+0000","last_active":"2023-10-14T11:58:16.296156+0000","last_peered":"2023-10-14T11:58:16.296156+0000","last_clean":"2023-10-14T11:58:16.296156+0000","last_became_active":"2023-10-14T11:58:15.169745+0000","last_became_peered":"2023-10-14T11:58:15.169745+0000","last_unstale":"2023-10-14T11:58:16.296156+0000","last_undegraded":"2023-10-14T11:58:16.296156+0000","last_fullsized":"2023-10-14T11:58:16.296156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:33:00.861183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296261+0000","last_change":"2023-10-14T11:58:14.151181+0000","last_active":"2023-10-14T11:58:16.296261+0000","last_peered":"2023-10-14T11:58:16.296261+0000","last_clean":"2023-10-14T11:58:16.296261+0000","last_became_active":"2023-10-14T11:58:14.150859+0000","last_became_peered":"2023-10-14T11:58:14.150859+0000","last_unstale":"2023-10-14T11:58:16.296261+0000","last_undegraded":"2023-10-14T11:58:16.296261+0000","last_fullsized":"2023-10-14T11:58:16.296261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:21:22.993387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144028+0000","last_change":"2023-10-14T11:58:15.168925+0000","last_active":"2023-10-14T11:58:16.144028+0000","last_peered":"2023-10-14T11:58:16.144028+0000","last_clean":"2023-10-14T11:58:16.144028+0000","last_became_active":"2023-10-14T11:58:15.168534+0000","last_became_peered":"2023-10-14T11:58:15.168534+0000","last_unstale":"2023-10-14T11:58:16.144028+0000","last_undegraded":"2023-10-14T11:58:16.144028+0000","last_fullsized":"2023-10-14T11:58:16.144028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:58:44.511366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144781+0000","last_change":"2023-10-14T11:58:14.150372+0000","last_active":"2023-10-14T11:58:16.144781+0000","last_peered":"2023-10-14T11:58:16.144781+0000","last_clean":"2023-10-14T11:58:16.144781+0000","last_became_active":"2023-10-14T11:58:14.150095+0000","last_became_peered":"2023-10-14T11:58:14.150095+0000","last_unstale":"2023-10-14T11:58:16.144781+0000","last_undegraded":"2023-10-14T11:58:16.144781+0000","last_fullsized":"2023-10-14T11:58:16.144781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:32:37.141853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298587+0000","last_change":"2023-10-14T11:58:15.158813+0000","last_active":"2023-10-14T11:58:16.298587+0000","last_peered":"2023-10-14T11:58:16.298587+0000","last_clean":"2023-10-14T11:58:16.298587+0000","last_became_active":"2023-10-14T11:58:15.158536+0000","last_became_peered":"2023-10-14T11:58:15.158536+0000","last_unstale":"2023-10-14T11:58:16.298587+0000","last_undegraded":"2023-10-14T11:58:16.298587+0000","last_fullsized":"2023-10-14T11:58:16.298587+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:25:39.003024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.145036+0000","last_change":"2023-10-14T11:58:14.153044+0000","last_active":"2023-10-14T11:58:16.145036+0000","last_peered":"2023-10-14T11:58:16.145036+0000","last_clean":"2023-10-14T11:58:16.145036+0000","last_became_active":"2023-10-14T11:58:14.152746+0000","last_became_peered":"2023-10-14T11:58:14.152746+0000","last_unstale":"2023-10-14T11:58:16.145036+0000","last_undegraded":"2023-10-14T11:58:16.145036+0000","last_fullsized":"2023-10-14T11:58:16.145036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:30:50.303297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298842+0000","last_change":"2023-10-14T11:58:15.174434+0000","last_active":"2023-10-14T11:58:16.298842+0000","last_peered":"2023-10-14T11:58:16.298842+0000","last_clean":"2023-10-14T11:58:16.298842+0000","last_became_active":"2023-10-14T11:58:15.174141+0000","last_became_peered":"2023-10-14T11:58:15.174141+0000","last_unstale":"2023-10-14T11:58:16.298842+0000","last_undegraded":"2023-10-14T11:58:16.298842+0000","last_fullsized":"2023-10-14T11:58:16.298842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:21:15.443882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296762+0000","last_change":"2023-10-14T11:58:14.149244+0000","last_active":"2023-10-14T11:58:16.296762+0000","last_peered":"2023-10-14T11:58:16.296762+0000","last_clean":"2023-10-14T11:58:16.296762+0000","last_became_active":"2023-10-14T11:58:14.148960+0000","last_became_peered":"2023-10-14T11:58:14.148960+0000","last_unstale":"2023-10-14T11:58:16.296762+0000","last_undegraded":"2023-10-14T11:58:16.296762+0000","last_fullsized":"2023-10-14T11:58:16.296762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:32:03.667630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144154+0000","last_change":"2023-10-14T11:58:15.150214+0000","last_active":"2023-10-14T11:58:16.144154+0000","last_peered":"2023-10-14T11:58:16.144154+0000","last_clean":"2023-10-14T11:58:16.144154+0000","last_became_active":"2023-10-14T11:58:15.149927+0000","last_became_peered":"2023-10-14T11:58:15.149927+0000","last_unstale":"2023-10-14T11:58:16.144154+0000","last_undegraded":"2023-10-14T11:58:16.144154+0000","last_fullsized":"2023-10-14T11:58:16.144154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:45:35.707554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298684+0000","last_change":"2023-10-14T11:58:14.153886+0000","last_active":"2023-10-14T11:58:16.298684+0000","last_peered":"2023-10-14T11:58:16.298684+0000","last_clean":"2023-10-14T11:58:16.298684+0000","last_became_active":"2023-10-14T11:58:14.153365+0000","last_became_peered":"2023-10-14T11:58:14.153365+0000","last_unstale":"2023-10-14T11:58:16.298684+0000","last_undegraded":"2023-10-14T11:58:16.298684+0000","last_fullsized":"2023-10-14T11:58:16.298684+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:21:22.993387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297257+0000","last_change":"2023-10-14T11:58:15.168448+0000","last_active":"2023-10-14T11:58:16.297257+0000","last_peered":"2023-10-14T11:58:16.297257+0000","last_clean":"2023-10-14T11:58:16.297257+0000","last_became_active":"2023-10-14T11:58:15.167757+0000","last_became_peered":"2023-10-14T11:58:15.167757+0000","last_unstale":"2023-10-14T11:58:16.297257+0000","last_undegraded":"2023-10-14T11:58:16.297257+0000","last_fullsized":"2023-10-14T11:58:16.297257+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:21:10.166149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.145150+0000","last_change":"2023-10-14T11:58:14.145505+0000","last_active":"2023-10-14T11:58:16.145150+0000","last_peered":"2023-10-14T11:58:16.145150+0000","last_clean":"2023-10-14T11:58:16.145150+0000","last_became_active":"2023-10-14T11:58:14.145218+0000","last_became_peered":"2023-10-14T11:58:14.145218+0000","last_unstale":"2023-10-14T11:58:16.145150+0000","last_undegraded":"2023-10-14T11:58:16.145150+0000","last_fullsized":"2023-10-14T11:58:16.145150+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:32:03.667630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297202+0000","last_change":"2023-10-14T11:58:15.162290+0000","last_active":"2023-10-14T11:58:16.297202+0000","last_peered":"2023-10-14T11:58:16.297202+0000","last_clean":"2023-10-14T11:58:16.297202+0000","last_became_active":"2023-10-14T11:58:15.161987+0000","last_became_peered":"2023-10-14T11:58:15.161987+0000","last_unstale":"2023-10-14T11:58:16.297202+0000","last_undegraded":"2023-10-14T11:58:16.297202+0000","last_fullsized":"2023-10-14T11:58:16.297202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:42:47.990050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298895+0000","last_change":"2023-10-14T11:58:14.154431+0000","last_active":"2023-10-14T11:58:16.298895+0000","last_peered":"2023-10-14T11:58:16.298895+0000","last_clean":"2023-10-14T11:58:16.298895+0000","last_became_active":"2023-10-14T11:58:14.153698+0000","last_became_peered":"2023-10-14T11:58:14.153698+0000","last_unstale":"2023-10-14T11:58:16.298895+0000","last_undegraded":"2023-10-14T11:58:16.298895+0000","last_fullsized":"2023-10-14T11:58:16.298895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:10:10.291507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297698+0000","last_change":"2023-10-14T11:58:15.160591+0000","last_active":"2023-10-14T11:58:16.297698+0000","last_peered":"2023-10-14T11:58:16.297698+0000","last_clean":"2023-10-14T11:58:16.297698+0000","last_became_active":"2023-10-14T11:58:15.160280+0000","last_became_peered":"2023-10-14T11:58:15.160280+0000","last_unstale":"2023-10-14T11:58:16.297698+0000","last_undegraded":"2023-10-14T11:58:16.297698+0000","last_fullsized":"2023-10-14T11:58:16.297698+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:42:47.990050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.143334+0000","last_change":"2023-10-14T11:58:15.156633+0000","last_active":"2023-10-14T11:58:16.143334+0000","last_peered":"2023-10-14T11:58:16.143334+0000","last_clean":"2023-10-14T11:58:16.143334+0000","last_became_active":"2023-10-14T11:58:15.156176+0000","last_became_peered":"2023-10-14T11:58:15.156176+0000","last_unstale":"2023-10-14T11:58:16.143334+0000","last_undegraded":"2023-10-14T11:58:16.143334+0000","last_fullsized":"2023-10-14T11:58:16.143334+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:25:39.003024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298424+0000","last_change":"2023-10-14T11:58:15.165873+0000","last_active":"2023-10-14T11:58:16.298424+0000","last_peered":"2023-10-14T11:58:16.298424+0000","last_clean":"2023-10-14T11:58:16.298424+0000","last_became_active":"2023-10-14T11:58:15.165592+0000","last_became_peered":"2023-10-14T11:58:15.165592+0000","last_unstale":"2023-10-14T11:58:16.298424+0000","last_undegraded":"2023-10-14T11:58:16.298424+0000","last_fullsized":"2023-10-14T11:58:16.298424+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:32:04.671265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296749+0000","last_change":"2023-10-14T11:58:15.165467+0000","last_active":"2023-10-14T11:58:16.296749+0000","last_peered":"2023-10-14T11:58:16.296749+0000","last_clean":"2023-10-14T11:58:16.296749+0000","last_became_active":"2023-10-14T11:58:15.165032+0000","last_became_peered":"2023-10-14T11:58:15.165032+0000","last_unstale":"2023-10-14T11:58:16.296749+0000","last_undegraded":"2023-10-14T11:58:16.296749+0000","last_fullsized":"2023-10-14T11:58:16.296749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:35:58.025887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296067+0000","last_change":"2023-10-14T11:58:15.170357+0000","last_active":"2023-10-14T11:58:16.296067+0000","last_peered":"2023-10-14T11:58:16.296067+0000","last_clean":"2023-10-14T11:58:16.296067+0000","last_became_active":"2023-10-14T11:58:15.170062+0000","last_became_peered":"2023-10-14T11:58:15.170062+0000","last_unstale":"2023-10-14T11:58:16.296067+0000","last_undegraded":"2023-10-14T11:58:16.296067+0000","last_fullsized":"2023-10-14T11:58:16.296067+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:25:39.003024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.299070+0000","last_change":"2023-10-14T11:58:15.156230+0000","last_active":"2023-10-14T11:58:16.299070+0000","last_peered":"2023-10-14T11:58:16.299070+0000","last_clean":"2023-10-14T11:58:16.299070+0000","last_became_active":"2023-10-14T11:58:15.155970+0000","last_became_peered":"2023-10-14T11:58:15.155970+0000","last_unstale":"2023-10-14T11:58:16.299070+0000","last_undegraded":"2023-10-14T11:58:16.299070+0000","last_fullsized":"2023-10-14T11:58:16.299070+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:15:07.231894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.145083+0000","last_change":"2023-10-14T11:58:15.158350+0000","last_active":"2023-10-14T11:58:16.145083+0000","last_peered":"2023-10-14T11:58:16.145083+0000","last_clean":"2023-10-14T11:58:16.145083+0000","last_became_active":"2023-10-14T11:58:15.158036+0000","last_became_peered":"2023-10-14T11:58:15.158036+0000","last_unstale":"2023-10-14T11:58:16.145083+0000","last_undegraded":"2023-10-14T11:58:16.145083+0000","last_fullsized":"2023-10-14T11:58:16.145083+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:21:15.443882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144612+0000","last_change":"2023-10-14T11:58:15.160864+0000","last_active":"2023-10-14T11:58:16.144612+0000","last_peered":"2023-10-14T11:58:16.144612+0000","last_clean":"2023-10-14T11:58:16.144612+0000","last_became_active":"2023-10-14T11:58:15.157887+0000","last_became_peered":"2023-10-14T11:58:15.157887+0000","last_unstale":"2023-10-14T11:58:16.144612+0000","last_undegraded":"2023-10-14T11:58:16.144612+0000","last_fullsized":"2023-10-14T11:58:16.144612+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:23:17.281352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.299205+0000","last_change":"2023-10-14T11:58:15.148543+0000","last_active":"2023-10-14T11:58:16.299205+0000","last_peered":"2023-10-14T11:58:16.299205+0000","last_clean":"2023-10-14T11:58:16.299205+0000","last_became_active":"2023-10-14T11:58:15.148273+0000","last_became_peered":"2023-10-14T11:58:15.148273+0000","last_unstale":"2023-10-14T11:58:16.299205+0000","last_undegraded":"2023-10-14T11:58:16.299205+0000","last_fullsized":"2023-10-14T11:58:16.299205+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:35:58.025887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296512+0000","last_change":"2023-10-14T11:58:15.162159+0000","last_active":"2023-10-14T11:58:16.296512+0000","last_peered":"2023-10-14T11:58:16.296512+0000","last_clean":"2023-10-14T11:58:16.296512+0000","last_became_active":"2023-10-14T11:58:15.161859+0000","last_became_peered":"2023-10-14T11:58:15.161859+0000","last_unstale":"2023-10-14T11:58:16.296512+0000","last_undegraded":"2023-10-14T11:58:16.296512+0000","last_fullsized":"2023-10-14T11:58:16.296512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:59:03.464936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298805+0000","last_change":"2023-10-14T11:58:15.147368+0000","last_active":"2023-10-14T11:58:16.298805+0000","last_peered":"2023-10-14T11:58:16.298805+0000","last_clean":"2023-10-14T11:58:16.298805+0000","last_became_active":"2023-10-14T11:58:15.147083+0000","last_became_peered":"2023-10-14T11:58:15.147083+0000","last_unstale":"2023-10-14T11:58:16.298805+0000","last_undegraded":"2023-10-14T11:58:16.298805+0000","last_fullsized":"2023-10-14T11:58:16.298805+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:01:03.783116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296598+0000","last_change":"2023-10-14T11:58:15.170122+0000","last_active":"2023-10-14T11:58:16.296598+0000","last_peered":"2023-10-14T11:58:16.296598+0000","last_clean":"2023-10-14T11:58:16.296598+0000","last_became_active":"2023-10-14T11:58:15.169483+0000","last_became_peered":"2023-10-14T11:58:15.169483+0000","last_unstale":"2023-10-14T11:58:16.296598+0000","last_undegraded":"2023-10-14T11:58:16.296598+0000","last_fullsized":"2023-10-14T11:58:16.296598+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:15:07.231894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296034+0000","last_change":"2023-10-14T11:58:15.170506+0000","last_active":"2023-10-14T11:58:16.296034+0000","last_peered":"2023-10-14T11:58:16.296034+0000","last_clean":"2023-10-14T11:58:16.296034+0000","last_became_active":"2023-10-14T11:58:15.170243+0000","last_became_peered":"2023-10-14T11:58:15.170243+0000","last_unstale":"2023-10-14T11:58:16.296034+0000","last_undegraded":"2023-10-14T11:58:16.296034+0000","last_fullsized":"2023-10-14T11:58:16.296034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:30:51.306932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144532+0000","last_change":"2023-10-14T11:58:15.147517+0000","last_active":"2023-10-14T11:58:16.144532+0000","last_peered":"2023-10-14T11:58:16.144532+0000","last_clean":"2023-10-14T11:58:16.144532+0000","last_became_active":"2023-10-14T11:58:15.147245+0000","last_became_peered":"2023-10-14T11:58:15.147245+0000","last_unstale":"2023-10-14T11:58:16.144532+0000","last_undegraded":"2023-10-14T11:58:16.144532+0000","last_fullsized":"2023-10-14T11:58:16.144532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:21:15.443882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65244,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":286720,"data_stored":666519,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2789999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":75729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83399999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":666575,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28100000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.752}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.218}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.226}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.3120000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.984000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2560000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.1589999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.035}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.6159999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.637}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.7160000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.7549999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T11:58:23.965 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T11:58:24.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.118+0000 7f33316d1700 1 Processor -- start 2023-10-14T11:58:24.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.118+0000 7f33316d1700 1 -- start start 2023-10-14T11:58:24.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.119+0000 7f33316d1700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f332c14b7e0 0x7f332c0c5150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:24.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.119+0000 7f33316d1700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f332c154b90 0x7f332c0c9260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:24.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.119+0000 7f33316d1700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f332c0c97a0 0x7f332c06b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:24.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.119+0000 7f33316d1700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f332c0c4e40 con 0x7f332c14b7e0 2023-10-14T11:58:24.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.119+0000 7f33316d1700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f332c0c81c0 con 0x7f332c0c97a0 2023-10-14T11:58:24.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.119+0000 7f33316d1700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f332c0c84a0 con 0x7f332c154b90 2023-10-14T11:58:24.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.119+0000 7f332affd700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f332c14b7e0 0x7f332c0c5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:24.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.119+0000 7f332affd700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f332c14b7e0 0x7f332c0c5150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44930/0 (socket says 172.21.15.78:44930) 2023-10-14T11:58:24.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.119+0000 7f332affd700 1 -- 172.21.15.78:0/2374637051 learned_addr learned my addr 172.21.15.78:0/2374637051 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:24.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.119+0000 7f332affd700 1 -- 172.21.15.78:0/2374637051 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f332c154b90 msgr2=0x7f332c0c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:24.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.119+0000 7f332affd700 1 --2- 172.21.15.78:0/2374637051 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f332c154b90 0x7f332c0c9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:24.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.119+0000 7f332affd700 1 -- 172.21.15.78:0/2374637051 >> v2:172.21.15.195:3300/0 conn(0x7f332c0c97a0 msgr2=0x7f332c06b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:24.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.119+0000 7f332b7fe700 1 --2- 172.21.15.78:0/2374637051 >> v2:172.21.15.195:3300/0 conn(0x7f332c0c97a0 0x7f332c06b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:24.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.119+0000 7f332affd700 1 --2- 172.21.15.78:0/2374637051 >> v2:172.21.15.195:3300/0 conn(0x7f332c0c97a0 0x7f332c06b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:24.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.119+0000 7f332affd700 1 -- 172.21.15.78:0/2374637051 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f332c0c8780 con 0x7f332c14b7e0 2023-10-14T11:58:24.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.119+0000 7f332b7fe700 1 --2- 172.21.15.78:0/2374637051 >> v2:172.21.15.195:3300/0 conn(0x7f332c0c97a0 0x7f332c06b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:24.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.119+0000 7f332affd700 1 --2- 172.21.15.78:0/2374637051 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f332c14b7e0 0x7f332c0c5150 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f332c14bbf0 tx=0x7f3320009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad346842e2832b39 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:24.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.119+0000 7f3329ffb700 1 -- 172.21.15.78:0/2374637051 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f332000b870 con 0x7f332c14b7e0 2023-10-14T11:58:24.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.119+0000 7f3329ffb700 1 -- 172.21.15.78:0/2374637051 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f332000b9d0 con 0x7f332c14b7e0 2023-10-14T11:58:24.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.120+0000 7f3329ffb700 1 -- 172.21.15.78:0/2374637051 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f332000bd60 con 0x7f332c14b7e0 2023-10-14T11:58:24.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.120+0000 7f33316d1700 1 -- 172.21.15.78:0/2374637051 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f332c14b7e0 msgr2=0x7f332c0c5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:24.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.120+0000 7f33316d1700 1 --2- 172.21.15.78:0/2374637051 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f332c14b7e0 0x7f332c0c5150 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f332c14bbf0 tx=0x7f3320009bd0 comp rx=0 tx=0).stop 2023-10-14T11:58:24.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.120+0000 7f33316d1700 1 -- 172.21.15.78:0/2374637051 shutdown_connections 2023-10-14T11:58:24.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.120+0000 7f33316d1700 1 --2- 172.21.15.78:0/2374637051 >> v2:172.21.15.195:3300/0 conn(0x7f332c0c97a0 0x7f332c06b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:24.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.120+0000 7f33316d1700 1 --2- 172.21.15.78:0/2374637051 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f332c14b7e0 0x7f332c0c5150 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:24.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.120+0000 7f33316d1700 1 --2- 172.21.15.78:0/2374637051 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f332c154b90 0x7f332c0c9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:24.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.120+0000 7f33316d1700 1 -- 172.21.15.78:0/2374637051 >> 172.21.15.78:0/2374637051 conn(0x7f332c0ca2c0 msgr2=0x7f332c0ca6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:24.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.120+0000 7f33316d1700 1 -- 172.21.15.78:0/2374637051 shutdown_connections 2023-10-14T11:58:24.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.120+0000 7f33316d1700 1 -- 172.21.15.78:0/2374637051 wait complete. 2023-10-14T11:58:24.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.121+0000 7f33316d1700 1 Processor -- start 2023-10-14T11:58:24.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.121+0000 7f33316d1700 1 -- start start 2023-10-14T11:58:24.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.122+0000 7f33316d1700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f332c14b7e0 0x7f332c0ae410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:24.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.122+0000 7f33316d1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f332c154b90 0x7f332c0ae950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:24.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.122+0000 7f33316d1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f332c147050 0x7f332c1474e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:24.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.122+0000 7f33316d1700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f332c0aee90 con 0x7f332c147050 2023-10-14T11:58:24.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.122+0000 7f33316d1700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f332c0afea0 con 0x7f332c154b90 2023-10-14T11:58:24.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.122+0000 7f33316d1700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f332c0c3780 con 0x7f332c14b7e0 2023-10-14T11:58:24.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.122+0000 7f332b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f332c147050 0x7f332c1474e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:24.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.122+0000 7f332b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f332c147050 0x7f332c1474e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44940/0 (socket says 172.21.15.78:44940) 2023-10-14T11:58:24.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.122+0000 7f332b7fe700 1 -- 172.21.15.78:0/3662447487 learned_addr learned my addr 172.21.15.78:0/3662447487 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:24.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.122+0000 7f332a7fc700 1 --2- 172.21.15.78:0/3662447487 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f332c154b90 0x7f332c0ae950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:24.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.122+0000 7f332b7fe700 1 -- 172.21.15.78:0/3662447487 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f332c14b7e0 msgr2=0x7f332c0ae410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:24.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.122+0000 7f332b7fe700 1 --2- 172.21.15.78:0/3662447487 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f332c14b7e0 0x7f332c0ae410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:24.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.122+0000 7f332b7fe700 1 -- 172.21.15.78:0/3662447487 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f332c154b90 msgr2=0x7f332c0ae950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:24.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.122+0000 7f332b7fe700 1 --2- 172.21.15.78:0/3662447487 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f332c154b90 0x7f332c0ae950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:24.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.122+0000 7f332b7fe700 1 -- 172.21.15.78:0/3662447487 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3320009550 con 0x7f332c147050 2023-10-14T11:58:24.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.122+0000 7f332b7fe700 1 --2- 172.21.15.78:0/3662447487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f332c147050 0x7f332c1474e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f332400ce40 tx=0x7f332400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:24.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.123+0000 7f33137fe700 1 -- 172.21.15.78:0/3662447487 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f332400adf0 con 0x7f332c147050 2023-10-14T11:58:24.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.123+0000 7f33137fe700 1 -- 172.21.15.78:0/3662447487 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3324003a20 con 0x7f332c147050 2023-10-14T11:58:24.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.123+0000 7f33316d1700 1 -- 172.21.15.78:0/3662447487 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f332c148b70 con 0x7f332c147050 2023-10-14T11:58:24.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.123+0000 7f33137fe700 1 -- 172.21.15.78:0/3662447487 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3324003dd0 con 0x7f332c147050 2023-10-14T11:58:24.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.123+0000 7f33316d1700 1 -- 172.21.15.78:0/3662447487 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f332c0c3a20 con 0x7f332c147050 2023-10-14T11:58:24.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.124+0000 7f33316d1700 1 -- 172.21.15.78:0/3662447487 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f332c0aac00 con 0x7f332c147050 2023-10-14T11:58:24.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.124+0000 7f33137fe700 1 -- 172.21.15.78:0/3662447487 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3324004000 con 0x7f332c147050 2023-10-14T11:58:24.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.124+0000 7f33137fe700 1 --2- 172.21.15.78:0/3662447487 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f331806af00 0x7f331806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:24.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.125+0000 7f332affd700 1 --2- 172.21.15.78:0/3662447487 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f331806af00 0x7f331806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:24.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.125+0000 7f33137fe700 1 -- 172.21.15.78:0/3662447487 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f33240898e0 con 0x7f332c147050 2023-10-14T11:58:24.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.125+0000 7f332affd700 1 --2- 172.21.15.78:0/3662447487 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f331806af00 0x7f331806d3c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3320009c10 tx=0x7f3320010000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:24.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.129+0000 7f33137fe700 1 -- 172.21.15.78:0/3662447487 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3324054a20 con 0x7f332c147050 2023-10-14T11:58:24.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.315+0000 7f33316d1700 1 -- 172.21.15.78:0/3662447487 --> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f332c0c8d40 con 0x7f331806af00 2023-10-14T11:58:24.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.322+0000 7f33137fe700 1 -- 172.21.15.78:0/3662447487 <== mgr.4104 v2:172.21.15.78:6832/3171890074 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264576 (secure 0 0 0) 0x7f332001f070 con 0x7f331806af00 2023-10-14T11:58:24.327 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:24.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.326+0000 7f33316d1700 1 -- 172.21.15.78:0/3662447487 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f331806af00 msgr2=0x7f331806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:24.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.326+0000 7f33316d1700 1 --2- 172.21.15.78:0/3662447487 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f331806af00 0x7f331806d3c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3320009c10 tx=0x7f3320010000 comp rx=0 tx=0).stop 2023-10-14T11:58:24.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.326+0000 7f33316d1700 1 -- 172.21.15.78:0/3662447487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f332c147050 msgr2=0x7f332c1474e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:24.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.326+0000 7f33316d1700 1 --2- 172.21.15.78:0/3662447487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f332c147050 0x7f332c1474e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f332400ce40 tx=0x7f332400cf20 comp rx=0 tx=0).stop 2023-10-14T11:58:24.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.326+0000 7f33316d1700 1 -- 172.21.15.78:0/3662447487 shutdown_connections 2023-10-14T11:58:24.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.326+0000 7f33316d1700 1 --2- 172.21.15.78:0/3662447487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f332c147050 0x7f332c1474e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:24.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.326+0000 7f33316d1700 1 --2- 172.21.15.78:0/3662447487 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f332c14b7e0 0x7f332c0ae410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:24.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.326+0000 7f33316d1700 1 --2- 172.21.15.78:0/3662447487 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f331806af00 0x7f331806d3c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:24.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.326+0000 7f33316d1700 1 --2- 172.21.15.78:0/3662447487 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f332c154b90 0x7f332c0ae950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:24.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.326+0000 7f33316d1700 1 -- 172.21.15.78:0/3662447487 >> 172.21.15.78:0/3662447487 conn(0x7f332c0ca2c0 msgr2=0x7f332c06b440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:24.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.326+0000 7f33316d1700 1 -- 172.21.15.78:0/3662447487 shutdown_connections 2023-10-14T11:58:24.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.326+0000 7f33316d1700 1 -- 172.21.15.78:0/3662447487 wait complete. 2023-10-14T11:58:24.332 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2023-10-14T11:58:24.340 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-14T11:58:22.966197+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218080,"kb_used_data":1888,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756640,"statfs":{"total":773094113280,"available":772870799360,"internally_reserved":0,"allocated":1933312,"data_stored":1790108,"data_compressed":65244,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.008217"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297796+0000","last_change":"2023-10-14T11:58:15.161449+0000","last_active":"2023-10-14T11:58:16.297796+0000","last_peered":"2023-10-14T11:58:16.297796+0000","last_clean":"2023-10-14T11:58:16.297796+0000","last_became_active":"2023-10-14T11:58:15.161100+0000","last_became_peered":"2023-10-14T11:58:15.161100+0000","last_unstale":"2023-10-14T11:58:16.297796+0000","last_undegraded":"2023-10-14T11:58:16.297796+0000","last_fullsized":"2023-10-14T11:58:16.297796+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:47:58.533238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.145107+0000","last_change":"2023-10-14T11:58:15.158210+0000","last_active":"2023-10-14T11:58:16.145107+0000","last_peered":"2023-10-14T11:58:16.145107+0000","last_clean":"2023-10-14T11:58:16.145107+0000","last_became_active":"2023-10-14T11:58:15.157886+0000","last_became_peered":"2023-10-14T11:58:15.157886+0000","last_unstale":"2023-10-14T11:58:16.145107+0000","last_undegraded":"2023-10-14T11:58:16.145107+0000","last_fullsized":"2023-10-14T11:58:16.145107+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:23:17.281352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296226+0000","last_change":"2023-10-14T11:58:15.161880+0000","last_active":"2023-10-14T11:58:16.296226+0000","last_peered":"2023-10-14T11:58:16.296226+0000","last_clean":"2023-10-14T11:58:16.296226+0000","last_became_active":"2023-10-14T11:58:15.161603+0000","last_became_peered":"2023-10-14T11:58:15.161603+0000","last_unstale":"2023-10-14T11:58:16.296226+0000","last_undegraded":"2023-10-14T11:58:16.296226+0000","last_fullsized":"2023-10-14T11:58:16.296226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:01:03.783116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296854+0000","last_change":"2023-10-14T11:58:15.167778+0000","last_active":"2023-10-14T11:58:16.296854+0000","last_peered":"2023-10-14T11:58:16.296854+0000","last_clean":"2023-10-14T11:58:16.296854+0000","last_became_active":"2023-10-14T11:58:15.167471+0000","last_became_peered":"2023-10-14T11:58:15.167471+0000","last_unstale":"2023-10-14T11:58:16.296854+0000","last_undegraded":"2023-10-14T11:58:16.296854+0000","last_fullsized":"2023-10-14T11:58:16.296854+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:07:58.884850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296653+0000","last_change":"2023-10-14T11:58:15.160788+0000","last_active":"2023-10-14T11:58:16.296653+0000","last_peered":"2023-10-14T11:58:16.296653+0000","last_clean":"2023-10-14T11:58:16.296653+0000","last_became_active":"2023-10-14T11:58:15.160525+0000","last_became_peered":"2023-10-14T11:58:15.160525+0000","last_unstale":"2023-10-14T11:58:16.296653+0000","last_undegraded":"2023-10-14T11:58:16.296653+0000","last_fullsized":"2023-10-14T11:58:16.296653+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:35:58.025887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297023+0000","last_change":"2023-10-14T11:58:15.167657+0000","last_active":"2023-10-14T11:58:16.297023+0000","last_peered":"2023-10-14T11:58:16.297023+0000","last_clean":"2023-10-14T11:58:16.297023+0000","last_became_active":"2023-10-14T11:58:15.167383+0000","last_became_peered":"2023-10-14T11:58:15.167383+0000","last_unstale":"2023-10-14T11:58:16.297023+0000","last_undegraded":"2023-10-14T11:58:16.297023+0000","last_fullsized":"2023-10-14T11:58:16.297023+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:53:19.926796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296834+0000","last_change":"2023-10-14T11:58:15.156244+0000","last_active":"2023-10-14T11:58:16.296834+0000","last_peered":"2023-10-14T11:58:16.296834+0000","last_clean":"2023-10-14T11:58:16.296834+0000","last_became_active":"2023-10-14T11:58:15.155621+0000","last_became_peered":"2023-10-14T11:58:15.155621+0000","last_unstale":"2023-10-14T11:58:16.296834+0000","last_undegraded":"2023-10-14T11:58:16.296834+0000","last_fullsized":"2023-10-14T11:58:16.296834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:07:58.884850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298542+0000","last_change":"2023-10-14T11:58:15.165865+0000","last_active":"2023-10-14T11:58:16.298542+0000","last_peered":"2023-10-14T11:58:16.298542+0000","last_clean":"2023-10-14T11:58:16.298542+0000","last_became_active":"2023-10-14T11:58:15.165613+0000","last_became_peered":"2023-10-14T11:58:15.165613+0000","last_unstale":"2023-10-14T11:58:16.298542+0000","last_undegraded":"2023-10-14T11:58:16.298542+0000","last_fullsized":"2023-10-14T11:58:16.298542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:32:38.145488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297728+0000","last_change":"2023-10-14T11:58:15.167047+0000","last_active":"2023-10-14T11:58:16.297728+0000","last_peered":"2023-10-14T11:58:16.297728+0000","last_clean":"2023-10-14T11:58:16.297728+0000","last_became_active":"2023-10-14T11:58:15.166362+0000","last_became_peered":"2023-10-14T11:58:15.166362+0000","last_unstale":"2023-10-14T11:58:16.297728+0000","last_undegraded":"2023-10-14T11:58:16.297728+0000","last_fullsized":"2023-10-14T11:58:16.297728+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:39:27.703491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144707+0000","last_change":"2023-10-14T11:58:15.164412+0000","last_active":"2023-10-14T11:58:16.144707+0000","last_peered":"2023-10-14T11:58:16.144707+0000","last_clean":"2023-10-14T11:58:16.144707+0000","last_became_active":"2023-10-14T11:58:15.164187+0000","last_became_peered":"2023-10-14T11:58:15.164187+0000","last_unstale":"2023-10-14T11:58:16.144707+0000","last_undegraded":"2023-10-14T11:58:16.144707+0000","last_fullsized":"2023-10-14T11:58:16.144707+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:59:03.464936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144466+0000","last_change":"2023-10-14T11:58:15.163857+0000","last_active":"2023-10-14T11:58:16.144466+0000","last_peered":"2023-10-14T11:58:16.144466+0000","last_clean":"2023-10-14T11:58:16.144466+0000","last_became_active":"2023-10-14T11:58:15.163575+0000","last_became_peered":"2023-10-14T11:58:15.163575+0000","last_unstale":"2023-10-14T11:58:16.144466+0000","last_undegraded":"2023-10-14T11:58:16.144466+0000","last_fullsized":"2023-10-14T11:58:16.144466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:01:03.783116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144222+0000","last_change":"2023-10-14T11:58:15.149569+0000","last_active":"2023-10-14T11:58:16.144222+0000","last_peered":"2023-10-14T11:58:16.144222+0000","last_clean":"2023-10-14T11:58:16.144222+0000","last_became_active":"2023-10-14T11:58:15.149161+0000","last_became_peered":"2023-10-14T11:58:15.149161+0000","last_unstale":"2023-10-14T11:58:16.144222+0000","last_undegraded":"2023-10-14T11:58:16.144222+0000","last_fullsized":"2023-10-14T11:58:16.144222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:19:30.143095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.299094+0000","last_change":"2023-10-14T11:58:15.147173+0000","last_active":"2023-10-14T11:58:16.299094+0000","last_peered":"2023-10-14T11:58:16.299094+0000","last_clean":"2023-10-14T11:58:16.299094+0000","last_became_active":"2023-10-14T11:58:15.146910+0000","last_became_peered":"2023-10-14T11:58:15.146910+0000","last_unstale":"2023-10-14T11:58:16.299094+0000","last_undegraded":"2023-10-14T11:58:16.299094+0000","last_fullsized":"2023-10-14T11:58:16.299094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:59:03.464936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296040+0000","last_change":"2023-10-14T11:58:15.169906+0000","last_active":"2023-10-14T11:58:16.296040+0000","last_peered":"2023-10-14T11:58:16.296040+0000","last_clean":"2023-10-14T11:58:16.296040+0000","last_became_active":"2023-10-14T11:58:15.169660+0000","last_became_peered":"2023-10-14T11:58:15.169660+0000","last_unstale":"2023-10-14T11:58:16.296040+0000","last_undegraded":"2023-10-14T11:58:16.296040+0000","last_fullsized":"2023-10-14T11:58:16.296040+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:32:38.145488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144820+0000","last_change":"2023-10-14T11:58:15.159353+0000","last_active":"2023-10-14T11:58:16.144820+0000","last_peered":"2023-10-14T11:58:16.144820+0000","last_clean":"2023-10-14T11:58:16.144820+0000","last_became_active":"2023-10-14T11:58:15.159095+0000","last_became_peered":"2023-10-14T11:58:15.159095+0000","last_unstale":"2023-10-14T11:58:16.144820+0000","last_undegraded":"2023-10-14T11:58:16.144820+0000","last_fullsized":"2023-10-14T11:58:16.144820+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:33:00.861183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296349+0000","last_change":"2023-10-14T11:58:15.169061+0000","last_active":"2023-10-14T11:58:16.296349+0000","last_peered":"2023-10-14T11:58:16.296349+0000","last_clean":"2023-10-14T11:58:16.296349+0000","last_became_active":"2023-10-14T11:58:15.168627+0000","last_became_peered":"2023-10-14T11:58:15.168627+0000","last_unstale":"2023-10-14T11:58:16.296349+0000","last_undegraded":"2023-10-14T11:58:16.296349+0000","last_fullsized":"2023-10-14T11:58:16.296349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:23:17.281352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297017+0000","last_change":"2023-10-14T11:58:15.162773+0000","last_active":"2023-10-14T11:58:16.297017+0000","last_peered":"2023-10-14T11:58:16.297017+0000","last_clean":"2023-10-14T11:58:16.297017+0000","last_became_active":"2023-10-14T11:58:15.162514+0000","last_became_peered":"2023-10-14T11:58:15.162514+0000","last_unstale":"2023-10-14T11:58:16.297017+0000","last_undegraded":"2023-10-14T11:58:16.297017+0000","last_fullsized":"2023-10-14T11:58:16.297017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:02:52.001714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298805+0000","last_change":"2023-10-14T11:58:15.164541+0000","last_active":"2023-10-14T11:58:16.298805+0000","last_peered":"2023-10-14T11:58:16.298805+0000","last_clean":"2023-10-14T11:58:16.298805+0000","last_became_active":"2023-10-14T11:58:15.164309+0000","last_became_peered":"2023-10-14T11:58:15.164309+0000","last_unstale":"2023-10-14T11:58:16.298805+0000","last_undegraded":"2023-10-14T11:58:16.298805+0000","last_fullsized":"2023-10-14T11:58:16.298805+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:22:04.803877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296451+0000","last_change":"2023-10-14T11:58:15.162171+0000","last_active":"2023-10-14T11:58:16.296451+0000","last_peered":"2023-10-14T11:58:16.296451+0000","last_clean":"2023-10-14T11:58:16.296451+0000","last_became_active":"2023-10-14T11:58:15.161894+0000","last_became_peered":"2023-10-14T11:58:15.161894+0000","last_unstale":"2023-10-14T11:58:16.296451+0000","last_undegraded":"2023-10-14T11:58:16.296451+0000","last_fullsized":"2023-10-14T11:58:16.296451+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:15:07.231894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296298+0000","last_change":"2023-10-14T11:58:14.145266+0000","last_active":"2023-10-14T11:58:16.296298+0000","last_peered":"2023-10-14T11:58:16.296298+0000","last_clean":"2023-10-14T11:58:16.296298+0000","last_became_active":"2023-10-14T11:58:14.144885+0000","last_became_peered":"2023-10-14T11:58:14.144885+0000","last_unstale":"2023-10-14T11:58:16.296298+0000","last_undegraded":"2023-10-14T11:58:16.296298+0000","last_fullsized":"2023-10-14T11:58:16.296298+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:06:27.985367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.299066+0000","last_change":"2023-10-14T11:58:15.148017+0000","last_active":"2023-10-14T11:58:16.299066+0000","last_peered":"2023-10-14T11:58:16.299066+0000","last_clean":"2023-10-14T11:58:16.299066+0000","last_became_active":"2023-10-14T11:58:15.147796+0000","last_became_peered":"2023-10-14T11:58:15.147796+0000","last_unstale":"2023-10-14T11:58:16.299066+0000","last_undegraded":"2023-10-14T11:58:16.299066+0000","last_fullsized":"2023-10-14T11:58:16.299066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:23:17.281352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.299129+0000","last_change":"2023-10-14T11:58:14.152335+0000","last_active":"2023-10-14T11:58:16.299129+0000","last_peered":"2023-10-14T11:58:16.299129+0000","last_clean":"2023-10-14T11:58:16.299129+0000","last_became_active":"2023-10-14T11:58:14.152064+0000","last_became_peered":"2023-10-14T11:58:14.152064+0000","last_unstale":"2023-10-14T11:58:16.299129+0000","last_undegraded":"2023-10-14T11:58:16.299129+0000","last_fullsized":"2023-10-14T11:58:16.299129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:21:22.993387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296225+0000","last_change":"2023-10-14T11:58:15.169510+0000","last_active":"2023-10-14T11:58:16.296225+0000","last_peered":"2023-10-14T11:58:16.296225+0000","last_clean":"2023-10-14T11:58:16.296225+0000","last_became_active":"2023-10-14T11:58:15.166488+0000","last_became_peered":"2023-10-14T11:58:15.166488+0000","last_unstale":"2023-10-14T11:58:16.296225+0000","last_undegraded":"2023-10-14T11:58:16.296225+0000","last_fullsized":"2023-10-14T11:58:16.296225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:23:17.281352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297447+0000","last_change":"2023-10-14T11:58:14.150333+0000","last_active":"2023-10-14T11:58:16.297447+0000","last_peered":"2023-10-14T11:58:16.297447+0000","last_clean":"2023-10-14T11:58:16.297447+0000","last_became_active":"2023-10-14T11:58:14.149979+0000","last_became_peered":"2023-10-14T11:58:14.149979+0000","last_unstale":"2023-10-14T11:58:16.297447+0000","last_undegraded":"2023-10-14T11:58:16.297447+0000","last_fullsized":"2023-10-14T11:58:16.297447+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:23:16.277717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297176+0000","last_change":"2023-10-14T11:58:15.163307+0000","last_active":"2023-10-14T11:58:16.297176+0000","last_peered":"2023-10-14T11:58:16.297176+0000","last_clean":"2023-10-14T11:58:16.297176+0000","last_became_active":"2023-10-14T11:58:15.163015+0000","last_became_peered":"2023-10-14T11:58:15.163015+0000","last_unstale":"2023-10-14T11:58:16.297176+0000","last_undegraded":"2023-10-14T11:58:16.297176+0000","last_fullsized":"2023-10-14T11:58:16.297176+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:21:10.166149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296924+0000","last_change":"2023-10-14T11:58:14.150931+0000","last_active":"2023-10-14T11:58:16.296924+0000","last_peered":"2023-10-14T11:58:16.296924+0000","last_clean":"2023-10-14T11:58:16.296924+0000","last_became_active":"2023-10-14T11:58:14.150629+0000","last_became_peered":"2023-10-14T11:58:14.150629+0000","last_unstale":"2023-10-14T11:58:16.296924+0000","last_undegraded":"2023-10-14T11:58:16.296924+0000","last_fullsized":"2023-10-14T11:58:16.296924+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:22:03.800242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297270+0000","last_change":"2023-10-14T11:58:15.163629+0000","last_active":"2023-10-14T11:58:16.297270+0000","last_peered":"2023-10-14T11:58:16.297270+0000","last_clean":"2023-10-14T11:58:16.297270+0000","last_became_active":"2023-10-14T11:58:15.163043+0000","last_became_peered":"2023-10-14T11:58:15.163043+0000","last_unstale":"2023-10-14T11:58:16.297270+0000","last_undegraded":"2023-10-14T11:58:16.297270+0000","last_fullsized":"2023-10-14T11:58:16.297270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:39:27.703491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298703+0000","last_change":"2023-10-14T11:58:14.154507+0000","last_active":"2023-10-14T11:58:16.298703+0000","last_peered":"2023-10-14T11:58:16.298703+0000","last_clean":"2023-10-14T11:58:16.298703+0000","last_became_active":"2023-10-14T11:58:14.154202+0000","last_became_peered":"2023-10-14T11:58:14.154202+0000","last_unstale":"2023-10-14T11:58:16.298703+0000","last_undegraded":"2023-10-14T11:58:16.298703+0000","last_fullsized":"2023-10-14T11:58:16.298703+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:59:45.791176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.143513+0000","last_change":"2023-10-14T11:58:15.158639+0000","last_active":"2023-10-14T11:58:16.143513+0000","last_peered":"2023-10-14T11:58:16.143513+0000","last_clean":"2023-10-14T11:58:16.143513+0000","last_became_active":"2023-10-14T11:58:15.158402+0000","last_became_peered":"2023-10-14T11:58:15.158402+0000","last_unstale":"2023-10-14T11:58:16.143513+0000","last_undegraded":"2023-10-14T11:58:16.143513+0000","last_fullsized":"2023-10-14T11:58:16.143513+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:21:23.997022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144263+0000","last_change":"2023-10-14T11:58:14.139502+0000","last_active":"2023-10-14T11:58:16.144263+0000","last_peered":"2023-10-14T11:58:16.144263+0000","last_clean":"2023-10-14T11:58:16.144263+0000","last_became_active":"2023-10-14T11:58:14.139053+0000","last_became_peered":"2023-10-14T11:58:14.139053+0000","last_unstale":"2023-10-14T11:58:16.144263+0000","last_undegraded":"2023-10-14T11:58:16.144263+0000","last_fullsized":"2023-10-14T11:58:16.144263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:10:10.291507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296162+0000","last_change":"2023-10-14T11:58:15.169858+0000","last_active":"2023-10-14T11:58:16.296162+0000","last_peered":"2023-10-14T11:58:16.296162+0000","last_clean":"2023-10-14T11:58:16.296162+0000","last_became_active":"2023-10-14T11:58:15.166724+0000","last_became_peered":"2023-10-14T11:58:15.166724+0000","last_unstale":"2023-10-14T11:58:16.296162+0000","last_undegraded":"2023-10-14T11:58:16.296162+0000","last_fullsized":"2023-10-14T11:58:16.296162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:45:35.707554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.299064+0000","last_change":"2023-10-14T11:58:14.154253+0000","last_active":"2023-10-14T11:58:16.299064+0000","last_peered":"2023-10-14T11:58:16.299064+0000","last_clean":"2023-10-14T11:58:16.299064+0000","last_became_active":"2023-10-14T11:58:14.152400+0000","last_became_peered":"2023-10-14T11:58:14.152400+0000","last_unstale":"2023-10-14T11:58:16.299064+0000","last_undegraded":"2023-10-14T11:58:16.299064+0000","last_fullsized":"2023-10-14T11:58:16.299064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:03:15.889637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298935+0000","last_change":"2023-10-14T11:58:15.167532+0000","last_active":"2023-10-14T11:58:16.298935+0000","last_peered":"2023-10-14T11:58:16.298935+0000","last_clean":"2023-10-14T11:58:16.298935+0000","last_became_active":"2023-10-14T11:58:15.167270+0000","last_became_peered":"2023-10-14T11:58:15.167270+0000","last_unstale":"2023-10-14T11:58:16.298935+0000","last_undegraded":"2023-10-14T11:58:16.298935+0000","last_fullsized":"2023-10-14T11:58:16.298935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:33:00.861183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296354+0000","last_change":"2023-10-14T11:58:14.147092+0000","last_active":"2023-10-14T11:58:16.296354+0000","last_peered":"2023-10-14T11:58:16.296354+0000","last_clean":"2023-10-14T11:58:16.296354+0000","last_became_active":"2023-10-14T11:58:14.146377+0000","last_became_peered":"2023-10-14T11:58:14.146377+0000","last_unstale":"2023-10-14T11:58:16.296354+0000","last_undegraded":"2023-10-14T11:58:16.296354+0000","last_fullsized":"2023-10-14T11:58:16.296354+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:25:37.999389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144924+0000","last_change":"2023-10-14T11:58:15.160286+0000","last_active":"2023-10-14T11:58:16.144924+0000","last_peered":"2023-10-14T11:58:16.144924+0000","last_clean":"2023-10-14T11:58:16.144924+0000","last_became_active":"2023-10-14T11:58:15.159308+0000","last_became_peered":"2023-10-14T11:58:15.159308+0000","last_unstale":"2023-10-14T11:58:16.144924+0000","last_undegraded":"2023-10-14T11:58:16.144924+0000","last_fullsized":"2023-10-14T11:58:16.144924+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:45:35.707554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144601+0000","last_change":"2023-10-14T11:58:14.148448+0000","last_active":"2023-10-14T11:58:16.144601+0000","last_peered":"2023-10-14T11:58:16.144601+0000","last_clean":"2023-10-14T11:58:16.144601+0000","last_became_active":"2023-10-14T11:58:14.148155+0000","last_became_peered":"2023-10-14T11:58:14.148155+0000","last_unstale":"2023-10-14T11:58:16.144601+0000","last_undegraded":"2023-10-14T11:58:16.144601+0000","last_fullsized":"2023-10-14T11:58:16.144601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:19:29.139460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296785+0000","last_change":"2023-10-14T11:58:15.166089+0000","last_active":"2023-10-14T11:58:16.296785+0000","last_peered":"2023-10-14T11:58:16.296785+0000","last_clean":"2023-10-14T11:58:16.296785+0000","last_became_active":"2023-10-14T11:58:15.164889+0000","last_became_peered":"2023-10-14T11:58:15.164889+0000","last_unstale":"2023-10-14T11:58:16.296785+0000","last_undegraded":"2023-10-14T11:58:16.296785+0000","last_fullsized":"2023-10-14T11:58:16.296785+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:15.482221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296897+0000","last_change":"2023-10-14T11:58:14.150455+0000","last_active":"2023-10-14T11:58:16.296897+0000","last_peered":"2023-10-14T11:58:16.296897+0000","last_clean":"2023-10-14T11:58:16.296897+0000","last_became_active":"2023-10-14T11:58:14.150195+0000","last_became_peered":"2023-10-14T11:58:14.150195+0000","last_unstale":"2023-10-14T11:58:16.296897+0000","last_undegraded":"2023-10-14T11:58:16.296897+0000","last_fullsized":"2023-10-14T11:58:16.296897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:32:59.857548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296229+0000","last_change":"2023-10-14T11:58:15.170459+0000","last_active":"2023-10-14T11:58:16.296229+0000","last_peered":"2023-10-14T11:58:16.296229+0000","last_clean":"2023-10-14T11:58:16.296229+0000","last_became_active":"2023-10-14T11:58:15.170096+0000","last_became_peered":"2023-10-14T11:58:15.170096+0000","last_unstale":"2023-10-14T11:58:16.296229+0000","last_undegraded":"2023-10-14T11:58:16.296229+0000","last_fullsized":"2023-10-14T11:58:16.296229+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:21:15.443882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.145168+0000","last_change":"2023-10-14T11:58:14.147259+0000","last_active":"2023-10-14T11:58:16.145168+0000","last_peered":"2023-10-14T11:58:16.145168+0000","last_clean":"2023-10-14T11:58:16.145168+0000","last_became_active":"2023-10-14T11:58:14.146982+0000","last_became_peered":"2023-10-14T11:58:14.146982+0000","last_unstale":"2023-10-14T11:58:16.145168+0000","last_undegraded":"2023-10-14T11:58:16.145168+0000","last_fullsized":"2023-10-14T11:58:16.145168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:22:03.800242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144175+0000","last_change":"2023-10-14T11:58:15.158917+0000","last_active":"2023-10-14T11:58:16.144175+0000","last_peered":"2023-10-14T11:58:16.144175+0000","last_clean":"2023-10-14T11:58:16.144175+0000","last_became_active":"2023-10-14T11:58:15.158578+0000","last_became_peered":"2023-10-14T11:58:15.158578+0000","last_unstale":"2023-10-14T11:58:16.144175+0000","last_undegraded":"2023-10-14T11:58:16.144175+0000","last_fullsized":"2023-10-14T11:58:16.144175+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:06:28.989002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144380+0000","last_change":"2023-10-14T11:58:14.152849+0000","last_active":"2023-10-14T11:58:16.144380+0000","last_peered":"2023-10-14T11:58:16.144380+0000","last_clean":"2023-10-14T11:58:16.144380+0000","last_became_active":"2023-10-14T11:58:14.152544+0000","last_became_peered":"2023-10-14T11:58:14.152544+0000","last_unstale":"2023-10-14T11:58:16.144380+0000","last_undegraded":"2023-10-14T11:58:16.144380+0000","last_fullsized":"2023-10-14T11:58:16.144380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:30:50.303297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297109+0000","last_change":"2023-10-14T11:58:15.160605+0000","last_active":"2023-10-14T11:58:16.297109+0000","last_peered":"2023-10-14T11:58:16.297109+0000","last_clean":"2023-10-14T11:58:16.297109+0000","last_became_active":"2023-10-14T11:58:15.160323+0000","last_became_peered":"2023-10-14T11:58:15.160323+0000","last_unstale":"2023-10-14T11:58:16.297109+0000","last_undegraded":"2023-10-14T11:58:16.297109+0000","last_fullsized":"2023-10-14T11:58:16.297109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:53:19.926796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296162+0000","last_change":"2023-10-14T11:58:14.145911+0000","last_active":"2023-10-14T11:58:16.296162+0000","last_peered":"2023-10-14T11:58:16.296162+0000","last_clean":"2023-10-14T11:58:16.296162+0000","last_became_active":"2023-10-14T11:58:14.144322+0000","last_became_peered":"2023-10-14T11:58:14.144322+0000","last_unstale":"2023-10-14T11:58:16.296162+0000","last_undegraded":"2023-10-14T11:58:16.296162+0000","last_fullsized":"2023-10-14T11:58:16.296162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:32:03.667630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.295965+0000","last_change":"2023-10-14T11:58:15.165189+0000","last_active":"2023-10-14T11:58:16.295965+0000","last_peered":"2023-10-14T11:58:16.295965+0000","last_clean":"2023-10-14T11:58:16.295965+0000","last_became_active":"2023-10-14T11:58:15.164911+0000","last_became_peered":"2023-10-14T11:58:15.164911+0000","last_unstale":"2023-10-14T11:58:16.295965+0000","last_undegraded":"2023-10-14T11:58:16.295965+0000","last_fullsized":"2023-10-14T11:58:16.295965+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:10:17.502466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297270+0000","last_change":"2023-10-14T11:58:14.152736+0000","last_active":"2023-10-14T11:58:16.297270+0000","last_peered":"2023-10-14T11:58:16.297270+0000","last_clean":"2023-10-14T11:58:16.297270+0000","last_became_active":"2023-10-14T11:58:14.152401+0000","last_became_peered":"2023-10-14T11:58:14.152401+0000","last_unstale":"2023-10-14T11:58:16.297270+0000","last_undegraded":"2023-10-14T11:58:16.297270+0000","last_fullsized":"2023-10-14T11:58:16.297270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:01:02.779481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296948+0000","last_change":"2023-10-14T11:58:15.155365+0000","last_active":"2023-10-14T11:58:16.296948+0000","last_peered":"2023-10-14T11:58:16.296948+0000","last_clean":"2023-10-14T11:58:16.296948+0000","last_became_active":"2023-10-14T11:58:15.155102+0000","last_became_peered":"2023-10-14T11:58:15.155102+0000","last_unstale":"2023-10-14T11:58:16.296948+0000","last_undegraded":"2023-10-14T11:58:16.296948+0000","last_fullsized":"2023-10-14T11:58:16.296948+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:02:52.001714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.143938+0000","last_change":"2023-10-14T11:58:14.148862+0000","last_active":"2023-10-14T11:58:16.143938+0000","last_peered":"2023-10-14T11:58:16.143938+0000","last_clean":"2023-10-14T11:58:16.143938+0000","last_became_active":"2023-10-14T11:58:14.148409+0000","last_became_peered":"2023-10-14T11:58:14.148409+0000","last_unstale":"2023-10-14T11:58:16.143938+0000","last_undegraded":"2023-10-14T11:58:16.143938+0000","last_fullsized":"2023-10-14T11:58:16.143938+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:06:27.985367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.143490+0000","last_change":"2023-10-14T11:58:15.156428+0000","last_active":"2023-10-14T11:58:16.143490+0000","last_peered":"2023-10-14T11:58:16.143490+0000","last_clean":"2023-10-14T11:58:16.143490+0000","last_became_active":"2023-10-14T11:58:15.155818+0000","last_became_peered":"2023-10-14T11:58:15.155818+0000","last_unstale":"2023-10-14T11:58:16.143490+0000","last_undegraded":"2023-10-14T11:58:16.143490+0000","last_fullsized":"2023-10-14T11:58:16.143490+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:33:00.861183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298391+0000","last_change":"2023-10-14T11:58:14.149643+0000","last_active":"2023-10-14T11:58:16.298391+0000","last_peered":"2023-10-14T11:58:16.298391+0000","last_clean":"2023-10-14T11:58:16.298391+0000","last_became_active":"2023-10-14T11:58:14.149330+0000","last_became_peered":"2023-10-14T11:58:14.149330+0000","last_unstale":"2023-10-14T11:58:16.298391+0000","last_undegraded":"2023-10-14T11:58:16.298391+0000","last_fullsized":"2023-10-14T11:58:16.298391+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:14:56.067093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296718+0000","last_change":"2023-10-14T11:58:15.156412+0000","last_active":"2023-10-14T11:58:16.296718+0000","last_peered":"2023-10-14T11:58:16.296718+0000","last_clean":"2023-10-14T11:58:16.296718+0000","last_became_active":"2023-10-14T11:58:15.155951+0000","last_became_peered":"2023-10-14T11:58:15.155951+0000","last_unstale":"2023-10-14T11:58:16.296718+0000","last_undegraded":"2023-10-14T11:58:16.296718+0000","last_fullsized":"2023-10-14T11:58:16.296718+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:15.482221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.143832+0000","last_change":"2023-10-14T11:58:14.135648+0000","last_active":"2023-10-14T11:58:16.143832+0000","last_peered":"2023-10-14T11:58:16.143832+0000","last_clean":"2023-10-14T11:58:16.143832+0000","last_became_active":"2023-10-14T11:58:14.135104+0000","last_became_peered":"2023-10-14T11:58:14.135104+0000","last_unstale":"2023-10-14T11:58:16.143832+0000","last_undegraded":"2023-10-14T11:58:16.143832+0000","last_fullsized":"2023-10-14T11:58:16.143832+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:47:41.184305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297156+0000","last_change":"2023-10-14T11:58:15.157354+0000","last_active":"2023-10-14T11:58:16.297156+0000","last_peered":"2023-10-14T11:58:16.297156+0000","last_clean":"2023-10-14T11:58:16.297156+0000","last_became_active":"2023-10-14T11:58:15.157084+0000","last_became_peered":"2023-10-14T11:58:15.157084+0000","last_unstale":"2023-10-14T11:58:16.297156+0000","last_undegraded":"2023-10-14T11:58:16.297156+0000","last_fullsized":"2023-10-14T11:58:16.297156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:32:40.379634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144935+0000","last_change":"2023-10-14T11:58:14.148246+0000","last_active":"2023-10-14T11:58:16.144935+0000","last_peered":"2023-10-14T11:58:16.144935+0000","last_clean":"2023-10-14T11:58:16.144935+0000","last_became_active":"2023-10-14T11:58:14.147943+0000","last_became_peered":"2023-10-14T11:58:14.147943+0000","last_unstale":"2023-10-14T11:58:16.144935+0000","last_undegraded":"2023-10-14T11:58:16.144935+0000","last_fullsized":"2023-10-14T11:58:16.144935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:19:29.139460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297125+0000","last_change":"2023-10-14T11:58:15.162066+0000","last_active":"2023-10-14T11:58:16.297125+0000","last_peered":"2023-10-14T11:58:16.297125+0000","last_clean":"2023-10-14T11:58:16.297125+0000","last_became_active":"2023-10-14T11:58:15.161798+0000","last_became_peered":"2023-10-14T11:58:15.161798+0000","last_unstale":"2023-10-14T11:58:16.297125+0000","last_undegraded":"2023-10-14T11:58:16.297125+0000","last_fullsized":"2023-10-14T11:58:16.297125+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:32:40.379634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297269+0000","last_change":"2023-10-14T11:58:14.147024+0000","last_active":"2023-10-14T11:58:16.297269+0000","last_peered":"2023-10-14T11:58:16.297269+0000","last_clean":"2023-10-14T11:58:16.297269+0000","last_became_active":"2023-10-14T11:58:14.146675+0000","last_became_peered":"2023-10-14T11:58:14.146675+0000","last_unstale":"2023-10-14T11:58:16.297269+0000","last_undegraded":"2023-10-14T11:58:16.297269+0000","last_fullsized":"2023-10-14T11:58:16.297269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:45:34.703919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.143832+0000","last_change":"2023-10-14T11:58:15.157978+0000","last_active":"2023-10-14T11:58:16.143832+0000","last_peered":"2023-10-14T11:58:16.143832+0000","last_clean":"2023-10-14T11:58:16.143832+0000","last_became_active":"2023-10-14T11:58:15.157698+0000","last_became_peered":"2023-10-14T11:58:15.157698+0000","last_unstale":"2023-10-14T11:58:16.143832+0000","last_undegraded":"2023-10-14T11:58:16.143832+0000","last_fullsized":"2023-10-14T11:58:16.143832+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:10:11.295142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144794+0000","last_change":"2023-10-14T11:58:15.164985+0000","last_active":"2023-10-14T11:58:16.144794+0000","last_peered":"2023-10-14T11:58:16.144794+0000","last_clean":"2023-10-14T11:58:16.144794+0000","last_became_active":"2023-10-14T11:58:15.164355+0000","last_became_peered":"2023-10-14T11:58:15.164355+0000","last_unstale":"2023-10-14T11:58:16.144794+0000","last_undegraded":"2023-10-14T11:58:16.144794+0000","last_fullsized":"2023-10-14T11:58:16.144794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:25:39.003024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298382+0000","last_change":"2023-10-14T11:58:14.153483+0000","last_active":"2023-10-14T11:58:16.298382+0000","last_peered":"2023-10-14T11:58:16.298382+0000","last_clean":"2023-10-14T11:58:16.298382+0000","last_became_active":"2023-10-14T11:58:14.153174+0000","last_became_peered":"2023-10-14T11:58:14.153174+0000","last_unstale":"2023-10-14T11:58:16.298382+0000","last_undegraded":"2023-10-14T11:58:16.298382+0000","last_fullsized":"2023-10-14T11:58:16.298382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:06:27.985367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298697+0000","last_change":"2023-10-14T11:58:15.171141+0000","last_active":"2023-10-14T11:58:16.298697+0000","last_peered":"2023-10-14T11:58:16.298697+0000","last_clean":"2023-10-14T11:58:16.298697+0000","last_became_active":"2023-10-14T11:58:15.170853+0000","last_became_peered":"2023-10-14T11:58:15.170853+0000","last_unstale":"2023-10-14T11:58:16.298697+0000","last_undegraded":"2023-10-14T11:58:16.298697+0000","last_fullsized":"2023-10-14T11:58:16.298697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:45:35.707554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296962+0000","last_change":"2023-10-14T11:58:14.153660+0000","last_active":"2023-10-14T11:58:16.296962+0000","last_peered":"2023-10-14T11:58:16.296962+0000","last_clean":"2023-10-14T11:58:16.296962+0000","last_became_active":"2023-10-14T11:58:14.153419+0000","last_became_peered":"2023-10-14T11:58:14.153419+0000","last_unstale":"2023-10-14T11:58:16.296962+0000","last_undegraded":"2023-10-14T11:58:16.296962+0000","last_fullsized":"2023-10-14T11:58:16.296962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:06:27.985367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T11:58:15.149401+0000","last_change":"2023-10-14T11:58:10.616726+0000","last_active":"2023-10-14T11:58:15.149401+0000","last_peered":"2023-10-14T11:58:15.149401+0000","last_clean":"2023-10-14T11:58:15.149401+0000","last_became_active":"2023-10-14T11:58:10.615628+0000","last_became_peered":"2023-10-14T11:58:10.615628+0000","last_unstale":"2023-10-14T11:58:15.149401+0000","last_undegraded":"2023-10-14T11:58:15.149401+0000","last_fullsized":"2023-10-14T11:58:15.149401+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:09.001265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:09.001265+0000","last_clean_scrub_stamp":"2023-10-14T11:58:09.001265+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:22:50.496050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.295927+0000","last_change":"2023-10-14T11:58:15.173632+0000","last_active":"2023-10-14T11:58:16.295927+0000","last_peered":"2023-10-14T11:58:16.295927+0000","last_clean":"2023-10-14T11:58:16.295927+0000","last_became_active":"2023-10-14T11:58:15.172996+0000","last_became_peered":"2023-10-14T11:58:15.172996+0000","last_unstale":"2023-10-14T11:58:16.295927+0000","last_undegraded":"2023-10-14T11:58:16.295927+0000","last_fullsized":"2023-10-14T11:58:16.295927+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:22:04.803877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296824+0000","last_change":"2023-10-14T11:58:14.150078+0000","last_active":"2023-10-14T11:58:16.296824+0000","last_peered":"2023-10-14T11:58:16.296824+0000","last_clean":"2023-10-14T11:58:16.296824+0000","last_became_active":"2023-10-14T11:58:14.149788+0000","last_became_peered":"2023-10-14T11:58:14.149788+0000","last_unstale":"2023-10-14T11:58:16.296824+0000","last_undegraded":"2023-10-14T11:58:16.296824+0000","last_fullsized":"2023-10-14T11:58:16.296824+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:10:10.291507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297353+0000","last_change":"2023-10-14T11:58:14.151386+0000","last_active":"2023-10-14T11:58:16.297353+0000","last_peered":"2023-10-14T11:58:16.297353+0000","last_clean":"2023-10-14T11:58:16.297353+0000","last_became_active":"2023-10-14T11:58:14.151041+0000","last_became_peered":"2023-10-14T11:58:14.151041+0000","last_unstale":"2023-10-14T11:58:16.297353+0000","last_undegraded":"2023-10-14T11:58:16.297353+0000","last_fullsized":"2023-10-14T11:58:16.297353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:21:14.440247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144942+0000","last_change":"2023-10-14T11:58:15.168958+0000","last_active":"2023-10-14T11:58:16.144942+0000","last_peered":"2023-10-14T11:58:16.144942+0000","last_clean":"2023-10-14T11:58:16.144942+0000","last_became_active":"2023-10-14T11:58:15.164516+0000","last_became_peered":"2023-10-14T11:58:15.164516+0000","last_unstale":"2023-10-14T11:58:16.144942+0000","last_undegraded":"2023-10-14T11:58:16.144942+0000","last_fullsized":"2023-10-14T11:58:16.144942+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:01:03.783116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:17.142950+0000","last_change":"2023-10-14T11:58:14.148617+0000","last_active":"2023-10-14T11:58:17.142950+0000","last_peered":"2023-10-14T11:58:17.142950+0000","last_clean":"2023-10-14T11:58:17.142950+0000","last_became_active":"2023-10-14T11:58:14.148324+0000","last_became_peered":"2023-10-14T11:58:14.148324+0000","last_unstale":"2023-10-14T11:58:17.142950+0000","last_undegraded":"2023-10-14T11:58:17.142950+0000","last_fullsized":"2023-10-14T11:58:17.142950+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:10:16.498831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.299219+0000","last_change":"2023-10-14T11:58:15.162441+0000","last_active":"2023-10-14T11:58:16.299219+0000","last_peered":"2023-10-14T11:58:16.299219+0000","last_clean":"2023-10-14T11:58:16.299219+0000","last_became_active":"2023-10-14T11:58:15.162047+0000","last_became_peered":"2023-10-14T11:58:15.162047+0000","last_unstale":"2023-10-14T11:58:16.299219+0000","last_undegraded":"2023-10-14T11:58:16.299219+0000","last_fullsized":"2023-10-14T11:58:16.299219+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:15.482221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144522+0000","last_change":"2023-10-14T11:58:14.151993+0000","last_active":"2023-10-14T11:58:16.144522+0000","last_peered":"2023-10-14T11:58:16.144522+0000","last_clean":"2023-10-14T11:58:16.144522+0000","last_became_active":"2023-10-14T11:58:14.151721+0000","last_became_peered":"2023-10-14T11:58:14.151721+0000","last_unstale":"2023-10-14T11:58:16.144522+0000","last_undegraded":"2023-10-14T11:58:16.144522+0000","last_fullsized":"2023-10-14T11:58:16.144522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:21:22.993387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296156+0000","last_change":"2023-10-14T11:58:15.170037+0000","last_active":"2023-10-14T11:58:16.296156+0000","last_peered":"2023-10-14T11:58:16.296156+0000","last_clean":"2023-10-14T11:58:16.296156+0000","last_became_active":"2023-10-14T11:58:15.169745+0000","last_became_peered":"2023-10-14T11:58:15.169745+0000","last_unstale":"2023-10-14T11:58:16.296156+0000","last_undegraded":"2023-10-14T11:58:16.296156+0000","last_fullsized":"2023-10-14T11:58:16.296156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:33:00.861183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296261+0000","last_change":"2023-10-14T11:58:14.151181+0000","last_active":"2023-10-14T11:58:16.296261+0000","last_peered":"2023-10-14T11:58:16.296261+0000","last_clean":"2023-10-14T11:58:16.296261+0000","last_became_active":"2023-10-14T11:58:14.150859+0000","last_became_peered":"2023-10-14T11:58:14.150859+0000","last_unstale":"2023-10-14T11:58:16.296261+0000","last_undegraded":"2023-10-14T11:58:16.296261+0000","last_fullsized":"2023-10-14T11:58:16.296261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:21:22.993387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144028+0000","last_change":"2023-10-14T11:58:15.168925+0000","last_active":"2023-10-14T11:58:16.144028+0000","last_peered":"2023-10-14T11:58:16.144028+0000","last_clean":"2023-10-14T11:58:16.144028+0000","last_became_active":"2023-10-14T11:58:15.168534+0000","last_became_peered":"2023-10-14T11:58:15.168534+0000","last_unstale":"2023-10-14T11:58:16.144028+0000","last_undegraded":"2023-10-14T11:58:16.144028+0000","last_fullsized":"2023-10-14T11:58:16.144028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:58:44.511366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144781+0000","last_change":"2023-10-14T11:58:14.150372+0000","last_active":"2023-10-14T11:58:16.144781+0000","last_peered":"2023-10-14T11:58:16.144781+0000","last_clean":"2023-10-14T11:58:16.144781+0000","last_became_active":"2023-10-14T11:58:14.150095+0000","last_became_peered":"2023-10-14T11:58:14.150095+0000","last_unstale":"2023-10-14T11:58:16.144781+0000","last_undegraded":"2023-10-14T11:58:16.144781+0000","last_fullsized":"2023-10-14T11:58:16.144781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:32:37.141853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298587+0000","last_change":"2023-10-14T11:58:15.158813+0000","last_active":"2023-10-14T11:58:16.298587+0000","last_peered":"2023-10-14T11:58:16.298587+0000","last_clean":"2023-10-14T11:58:16.298587+0000","last_became_active":"2023-10-14T11:58:15.158536+0000","last_became_peered":"2023-10-14T11:58:15.158536+0000","last_unstale":"2023-10-14T11:58:16.298587+0000","last_undegraded":"2023-10-14T11:58:16.298587+0000","last_fullsized":"2023-10-14T11:58:16.298587+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:25:39.003024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.145036+0000","last_change":"2023-10-14T11:58:14.153044+0000","last_active":"2023-10-14T11:58:16.145036+0000","last_peered":"2023-10-14T11:58:16.145036+0000","last_clean":"2023-10-14T11:58:16.145036+0000","last_became_active":"2023-10-14T11:58:14.152746+0000","last_became_peered":"2023-10-14T11:58:14.152746+0000","last_unstale":"2023-10-14T11:58:16.145036+0000","last_undegraded":"2023-10-14T11:58:16.145036+0000","last_fullsized":"2023-10-14T11:58:16.145036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:30:50.303297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298842+0000","last_change":"2023-10-14T11:58:15.174434+0000","last_active":"2023-10-14T11:58:16.298842+0000","last_peered":"2023-10-14T11:58:16.298842+0000","last_clean":"2023-10-14T11:58:16.298842+0000","last_became_active":"2023-10-14T11:58:15.174141+0000","last_became_peered":"2023-10-14T11:58:15.174141+0000","last_unstale":"2023-10-14T11:58:16.298842+0000","last_undegraded":"2023-10-14T11:58:16.298842+0000","last_fullsized":"2023-10-14T11:58:16.298842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:21:15.443882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296762+0000","last_change":"2023-10-14T11:58:14.149244+0000","last_active":"2023-10-14T11:58:16.296762+0000","last_peered":"2023-10-14T11:58:16.296762+0000","last_clean":"2023-10-14T11:58:16.296762+0000","last_became_active":"2023-10-14T11:58:14.148960+0000","last_became_peered":"2023-10-14T11:58:14.148960+0000","last_unstale":"2023-10-14T11:58:16.296762+0000","last_undegraded":"2023-10-14T11:58:16.296762+0000","last_fullsized":"2023-10-14T11:58:16.296762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:32:03.667630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144154+0000","last_change":"2023-10-14T11:58:15.150214+0000","last_active":"2023-10-14T11:58:16.144154+0000","last_peered":"2023-10-14T11:58:16.144154+0000","last_clean":"2023-10-14T11:58:16.144154+0000","last_became_active":"2023-10-14T11:58:15.149927+0000","last_became_peered":"2023-10-14T11:58:15.149927+0000","last_unstale":"2023-10-14T11:58:16.144154+0000","last_undegraded":"2023-10-14T11:58:16.144154+0000","last_fullsized":"2023-10-14T11:58:16.144154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:45:35.707554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298684+0000","last_change":"2023-10-14T11:58:14.153886+0000","last_active":"2023-10-14T11:58:16.298684+0000","last_peered":"2023-10-14T11:58:16.298684+0000","last_clean":"2023-10-14T11:58:16.298684+0000","last_became_active":"2023-10-14T11:58:14.153365+0000","last_became_peered":"2023-10-14T11:58:14.153365+0000","last_unstale":"2023-10-14T11:58:16.298684+0000","last_undegraded":"2023-10-14T11:58:16.298684+0000","last_fullsized":"2023-10-14T11:58:16.298684+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:21:22.993387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297257+0000","last_change":"2023-10-14T11:58:15.168448+0000","last_active":"2023-10-14T11:58:16.297257+0000","last_peered":"2023-10-14T11:58:16.297257+0000","last_clean":"2023-10-14T11:58:16.297257+0000","last_became_active":"2023-10-14T11:58:15.167757+0000","last_became_peered":"2023-10-14T11:58:15.167757+0000","last_unstale":"2023-10-14T11:58:16.297257+0000","last_undegraded":"2023-10-14T11:58:16.297257+0000","last_fullsized":"2023-10-14T11:58:16.297257+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:21:10.166149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.145150+0000","last_change":"2023-10-14T11:58:14.145505+0000","last_active":"2023-10-14T11:58:16.145150+0000","last_peered":"2023-10-14T11:58:16.145150+0000","last_clean":"2023-10-14T11:58:16.145150+0000","last_became_active":"2023-10-14T11:58:14.145218+0000","last_became_peered":"2023-10-14T11:58:14.145218+0000","last_unstale":"2023-10-14T11:58:16.145150+0000","last_undegraded":"2023-10-14T11:58:16.145150+0000","last_fullsized":"2023-10-14T11:58:16.145150+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:32:03.667630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297202+0000","last_change":"2023-10-14T11:58:15.162290+0000","last_active":"2023-10-14T11:58:16.297202+0000","last_peered":"2023-10-14T11:58:16.297202+0000","last_clean":"2023-10-14T11:58:16.297202+0000","last_became_active":"2023-10-14T11:58:15.161987+0000","last_became_peered":"2023-10-14T11:58:15.161987+0000","last_unstale":"2023-10-14T11:58:16.297202+0000","last_undegraded":"2023-10-14T11:58:16.297202+0000","last_fullsized":"2023-10-14T11:58:16.297202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:42:47.990050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298895+0000","last_change":"2023-10-14T11:58:14.154431+0000","last_active":"2023-10-14T11:58:16.298895+0000","last_peered":"2023-10-14T11:58:16.298895+0000","last_clean":"2023-10-14T11:58:16.298895+0000","last_became_active":"2023-10-14T11:58:14.153698+0000","last_became_peered":"2023-10-14T11:58:14.153698+0000","last_unstale":"2023-10-14T11:58:16.298895+0000","last_undegraded":"2023-10-14T11:58:16.298895+0000","last_fullsized":"2023-10-14T11:58:16.298895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:13.121030+0000","last_clean_scrub_stamp":"2023-10-14T11:58:13.121030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:10:10.291507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.297698+0000","last_change":"2023-10-14T11:58:15.160591+0000","last_active":"2023-10-14T11:58:16.297698+0000","last_peered":"2023-10-14T11:58:16.297698+0000","last_clean":"2023-10-14T11:58:16.297698+0000","last_became_active":"2023-10-14T11:58:15.160280+0000","last_became_peered":"2023-10-14T11:58:15.160280+0000","last_unstale":"2023-10-14T11:58:16.297698+0000","last_undegraded":"2023-10-14T11:58:16.297698+0000","last_fullsized":"2023-10-14T11:58:16.297698+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:42:47.990050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.143334+0000","last_change":"2023-10-14T11:58:15.156633+0000","last_active":"2023-10-14T11:58:16.143334+0000","last_peered":"2023-10-14T11:58:16.143334+0000","last_clean":"2023-10-14T11:58:16.143334+0000","last_became_active":"2023-10-14T11:58:15.156176+0000","last_became_peered":"2023-10-14T11:58:15.156176+0000","last_unstale":"2023-10-14T11:58:16.143334+0000","last_undegraded":"2023-10-14T11:58:16.143334+0000","last_fullsized":"2023-10-14T11:58:16.143334+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:25:39.003024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298424+0000","last_change":"2023-10-14T11:58:15.165873+0000","last_active":"2023-10-14T11:58:16.298424+0000","last_peered":"2023-10-14T11:58:16.298424+0000","last_clean":"2023-10-14T11:58:16.298424+0000","last_became_active":"2023-10-14T11:58:15.165592+0000","last_became_peered":"2023-10-14T11:58:15.165592+0000","last_unstale":"2023-10-14T11:58:16.298424+0000","last_undegraded":"2023-10-14T11:58:16.298424+0000","last_fullsized":"2023-10-14T11:58:16.298424+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:32:04.671265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296749+0000","last_change":"2023-10-14T11:58:15.165467+0000","last_active":"2023-10-14T11:58:16.296749+0000","last_peered":"2023-10-14T11:58:16.296749+0000","last_clean":"2023-10-14T11:58:16.296749+0000","last_became_active":"2023-10-14T11:58:15.165032+0000","last_became_peered":"2023-10-14T11:58:15.165032+0000","last_unstale":"2023-10-14T11:58:16.296749+0000","last_undegraded":"2023-10-14T11:58:16.296749+0000","last_fullsized":"2023-10-14T11:58:16.296749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:35:58.025887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296067+0000","last_change":"2023-10-14T11:58:15.170357+0000","last_active":"2023-10-14T11:58:16.296067+0000","last_peered":"2023-10-14T11:58:16.296067+0000","last_clean":"2023-10-14T11:58:16.296067+0000","last_became_active":"2023-10-14T11:58:15.170062+0000","last_became_peered":"2023-10-14T11:58:15.170062+0000","last_unstale":"2023-10-14T11:58:16.296067+0000","last_undegraded":"2023-10-14T11:58:16.296067+0000","last_fullsized":"2023-10-14T11:58:16.296067+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:25:39.003024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.299070+0000","last_change":"2023-10-14T11:58:15.156230+0000","last_active":"2023-10-14T11:58:16.299070+0000","last_peered":"2023-10-14T11:58:16.299070+0000","last_clean":"2023-10-14T11:58:16.299070+0000","last_became_active":"2023-10-14T11:58:15.155970+0000","last_became_peered":"2023-10-14T11:58:15.155970+0000","last_unstale":"2023-10-14T11:58:16.299070+0000","last_undegraded":"2023-10-14T11:58:16.299070+0000","last_fullsized":"2023-10-14T11:58:16.299070+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:15:07.231894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.145083+0000","last_change":"2023-10-14T11:58:15.158350+0000","last_active":"2023-10-14T11:58:16.145083+0000","last_peered":"2023-10-14T11:58:16.145083+0000","last_clean":"2023-10-14T11:58:16.145083+0000","last_became_active":"2023-10-14T11:58:15.158036+0000","last_became_peered":"2023-10-14T11:58:15.158036+0000","last_unstale":"2023-10-14T11:58:16.145083+0000","last_undegraded":"2023-10-14T11:58:16.145083+0000","last_fullsized":"2023-10-14T11:58:16.145083+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:21:15.443882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144612+0000","last_change":"2023-10-14T11:58:15.160864+0000","last_active":"2023-10-14T11:58:16.144612+0000","last_peered":"2023-10-14T11:58:16.144612+0000","last_clean":"2023-10-14T11:58:16.144612+0000","last_became_active":"2023-10-14T11:58:15.157887+0000","last_became_peered":"2023-10-14T11:58:15.157887+0000","last_unstale":"2023-10-14T11:58:16.144612+0000","last_undegraded":"2023-10-14T11:58:16.144612+0000","last_fullsized":"2023-10-14T11:58:16.144612+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:23:17.281352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.299205+0000","last_change":"2023-10-14T11:58:15.148543+0000","last_active":"2023-10-14T11:58:16.299205+0000","last_peered":"2023-10-14T11:58:16.299205+0000","last_clean":"2023-10-14T11:58:16.299205+0000","last_became_active":"2023-10-14T11:58:15.148273+0000","last_became_peered":"2023-10-14T11:58:15.148273+0000","last_unstale":"2023-10-14T11:58:16.299205+0000","last_undegraded":"2023-10-14T11:58:16.299205+0000","last_fullsized":"2023-10-14T11:58:16.299205+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:35:58.025887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296512+0000","last_change":"2023-10-14T11:58:15.162159+0000","last_active":"2023-10-14T11:58:16.296512+0000","last_peered":"2023-10-14T11:58:16.296512+0000","last_clean":"2023-10-14T11:58:16.296512+0000","last_became_active":"2023-10-14T11:58:15.161859+0000","last_became_peered":"2023-10-14T11:58:15.161859+0000","last_unstale":"2023-10-14T11:58:16.296512+0000","last_undegraded":"2023-10-14T11:58:16.296512+0000","last_fullsized":"2023-10-14T11:58:16.296512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:59:03.464936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.298805+0000","last_change":"2023-10-14T11:58:15.147368+0000","last_active":"2023-10-14T11:58:16.298805+0000","last_peered":"2023-10-14T11:58:16.298805+0000","last_clean":"2023-10-14T11:58:16.298805+0000","last_became_active":"2023-10-14T11:58:15.147083+0000","last_became_peered":"2023-10-14T11:58:15.147083+0000","last_unstale":"2023-10-14T11:58:16.298805+0000","last_undegraded":"2023-10-14T11:58:16.298805+0000","last_fullsized":"2023-10-14T11:58:16.298805+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:01:03.783116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296598+0000","last_change":"2023-10-14T11:58:15.170122+0000","last_active":"2023-10-14T11:58:16.296598+0000","last_peered":"2023-10-14T11:58:16.296598+0000","last_clean":"2023-10-14T11:58:16.296598+0000","last_became_active":"2023-10-14T11:58:15.169483+0000","last_became_peered":"2023-10-14T11:58:15.169483+0000","last_unstale":"2023-10-14T11:58:16.296598+0000","last_undegraded":"2023-10-14T11:58:16.296598+0000","last_fullsized":"2023-10-14T11:58:16.296598+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:15:07.231894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.296034+0000","last_change":"2023-10-14T11:58:15.170506+0000","last_active":"2023-10-14T11:58:16.296034+0000","last_peered":"2023-10-14T11:58:16.296034+0000","last_clean":"2023-10-14T11:58:16.296034+0000","last_became_active":"2023-10-14T11:58:15.170243+0000","last_became_peered":"2023-10-14T11:58:15.170243+0000","last_unstale":"2023-10-14T11:58:16.296034+0000","last_undegraded":"2023-10-14T11:58:16.296034+0000","last_fullsized":"2023-10-14T11:58:16.296034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:30:51.306932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-14T11:58:16.144532+0000","last_change":"2023-10-14T11:58:15.147517+0000","last_active":"2023-10-14T11:58:16.144532+0000","last_peered":"2023-10-14T11:58:16.144532+0000","last_clean":"2023-10-14T11:58:16.144532+0000","last_became_active":"2023-10-14T11:58:15.147245+0000","last_became_peered":"2023-10-14T11:58:15.147245+0000","last_unstale":"2023-10-14T11:58:16.144532+0000","last_undegraded":"2023-10-14T11:58:16.144532+0000","last_fullsized":"2023-10-14T11:58:16.144532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:14.124665+0000","last_clean_scrub_stamp":"2023-10-14T11:58:14.124665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:21:15.443882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65244,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":286720,"data_stored":666519,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2789999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":75729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83399999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":666575,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28100000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.752}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.218}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.226}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.3120000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.984000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2560000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.1589999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.035}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.6159999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.637}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.7160000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.7549999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T11:58:24.345 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-14T11:58:24.345 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-14T11:58:24.346 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-14T11:58:24.346 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-14T11:58:24.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.498+0000 7f55b2002700 1 Processor -- start 2023-10-14T11:58:24.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.498+0000 7f55b2002700 1 -- start start 2023-10-14T11:58:24.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.498+0000 7f55b2002700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f55ac06ab40 0x7f55ac06f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:24.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.498+0000 7f55b2002700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f55ac1554e0 0x7f55ac06af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:24.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.498+0000 7f55b2002700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f55ac0c0350 0x7f55ac13d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:24.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.498+0000 7f55b2002700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55ac0c49f0 con 0x7f55ac06ab40 2023-10-14T11:58:24.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.498+0000 7f55b2002700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55ac0726a0 con 0x7f55ac0c0350 2023-10-14T11:58:24.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.498+0000 7f55b2002700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55ac072980 con 0x7f55ac1554e0 2023-10-14T11:58:24.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.498+0000 7f55ab7fe700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f55ac06ab40 0x7f55ac06f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:24.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.499+0000 7f55ab7fe700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f55ac06ab40 0x7f55ac06f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44944/0 (socket says 172.21.15.78:44944) 2023-10-14T11:58:24.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.499+0000 7f55ab7fe700 1 -- 172.21.15.78:0/650409963 learned_addr learned my addr 172.21.15.78:0/650409963 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:24.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.499+0000 7f55abfff700 1 --2- 172.21.15.78:0/650409963 >> v2:172.21.15.195:3300/0 conn(0x7f55ac0c0350 0x7f55ac13d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:24.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.499+0000 7f55aaffd700 1 --2- 172.21.15.78:0/650409963 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f55ac1554e0 0x7f55ac06af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:24.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.499+0000 7f55ab7fe700 1 -- 172.21.15.78:0/650409963 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f55ac1554e0 msgr2=0x7f55ac06af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:24.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.499+0000 7f55ab7fe700 1 --2- 172.21.15.78:0/650409963 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f55ac1554e0 0x7f55ac06af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:24.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.499+0000 7f55ab7fe700 1 -- 172.21.15.78:0/650409963 >> v2:172.21.15.195:3300/0 conn(0x7f55ac0c0350 msgr2=0x7f55ac13d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:24.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.499+0000 7f55ab7fe700 1 --2- 172.21.15.78:0/650409963 >> v2:172.21.15.195:3300/0 conn(0x7f55ac0c0350 0x7f55ac13d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:24.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.499+0000 7f55ab7fe700 1 -- 172.21.15.78:0/650409963 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55ac154a00 con 0x7f55ac06ab40 2023-10-14T11:58:24.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.499+0000 7f55aaffd700 1 --2- 172.21.15.78:0/650409963 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f55ac1554e0 0x7f55ac06af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:24.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.499+0000 7f55ab7fe700 1 --2- 172.21.15.78:0/650409963 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55ac06ab40 0x7f55ac06f1e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f55ac06c890 tx=0x7f559c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ab7c788c43d32d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:24.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.499+0000 7f55aa7fc700 1 -- 172.21.15.78:0/650409963 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559c00b870 con 0x7f55ac06ab40 2023-10-14T11:58:24.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.499+0000 7f55aa7fc700 1 -- 172.21.15.78:0/650409963 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f559c00b9d0 con 0x7f55ac06ab40 2023-10-14T11:58:24.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.499+0000 7f55aa7fc700 1 -- 172.21.15.78:0/650409963 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559c00bd60 con 0x7f55ac06ab40 2023-10-14T11:58:24.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.500+0000 7f55b2002700 1 -- 172.21.15.78:0/650409963 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55ac06ab40 msgr2=0x7f55ac06f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:24.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.500+0000 7f55b2002700 1 --2- 172.21.15.78:0/650409963 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55ac06ab40 0x7f55ac06f1e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f55ac06c890 tx=0x7f559c009bd0 comp rx=0 tx=0).stop 2023-10-14T11:58:24.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.500+0000 7f55b2002700 1 -- 172.21.15.78:0/650409963 shutdown_connections 2023-10-14T11:58:24.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.500+0000 7f55b2002700 1 --2- 172.21.15.78:0/650409963 >> v2:172.21.15.195:3300/0 conn(0x7f55ac0c0350 0x7f55ac13d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:24.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.500+0000 7f55b2002700 1 --2- 172.21.15.78:0/650409963 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55ac06ab40 0x7f55ac06f1e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:24.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.500+0000 7f55b2002700 1 --2- 172.21.15.78:0/650409963 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f55ac1554e0 0x7f55ac06af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:24.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.500+0000 7f55b2002700 1 -- 172.21.15.78:0/650409963 >> 172.21.15.78:0/650409963 conn(0x7f55ac0c3ac0 msgr2=0x7f55ac0c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:24.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.500+0000 7f55b2002700 1 -- 172.21.15.78:0/650409963 shutdown_connections 2023-10-14T11:58:24.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.500+0000 7f55b2002700 1 -- 172.21.15.78:0/650409963 wait complete. 2023-10-14T11:58:24.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.501+0000 7f55b2002700 1 Processor -- start 2023-10-14T11:58:24.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.501+0000 7f55b2002700 1 -- start start 2023-10-14T11:58:24.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.502+0000 7f55b2002700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55ac06ab40 0x7f55ac13b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:24.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.502+0000 7f55b2002700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f55ac0c0350 0x7f55ac134360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:24.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.502+0000 7f55b2002700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f55ac1554e0 0x7f55ac1348a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:24.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.502+0000 7f55b2002700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55ac1e7910 con 0x7f55ac06ab40 2023-10-14T11:58:24.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.502+0000 7f55b2002700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55ac1edbf0 con 0x7f55ac1554e0 2023-10-14T11:58:24.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.502+0000 7f55b2002700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55ac1ee120 con 0x7f55ac0c0350 2023-10-14T11:58:24.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.502+0000 7f55ab7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55ac06ab40 0x7f55ac13b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:24.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.502+0000 7f55ab7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55ac06ab40 0x7f55ac13b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44950/0 (socket says 172.21.15.78:44950) 2023-10-14T11:58:24.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.502+0000 7f55ab7fe700 1 -- 172.21.15.78:0/4071348369 learned_addr learned my addr 172.21.15.78:0/4071348369 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:24.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.502+0000 7f55aaffd700 1 --2- 172.21.15.78:0/4071348369 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f55ac0c0350 0x7f55ac134360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:24.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.502+0000 7f55abfff700 1 --2- 172.21.15.78:0/4071348369 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f55ac1554e0 0x7f55ac1348a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:24.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.502+0000 7f55ab7fe700 1 -- 172.21.15.78:0/4071348369 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f55ac0c0350 msgr2=0x7f55ac134360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:24.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.502+0000 7f55ab7fe700 1 --2- 172.21.15.78:0/4071348369 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f55ac0c0350 0x7f55ac134360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:24.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.502+0000 7f55ab7fe700 1 -- 172.21.15.78:0/4071348369 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f55ac1554e0 msgr2=0x7f55ac1348a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:24.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.502+0000 7f55ab7fe700 1 --2- 172.21.15.78:0/4071348369 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f55ac1554e0 0x7f55ac1348a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:24.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.502+0000 7f55ab7fe700 1 -- 172.21.15.78:0/4071348369 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f559c009550 con 0x7f55ac06ab40 2023-10-14T11:58:24.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.502+0000 7f55ab7fe700 1 --2- 172.21.15.78:0/4071348369 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55ac06ab40 0x7f55ac13b6c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f559c000c00 tx=0x7f559c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:24.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.502+0000 7f5593fff700 1 -- 172.21.15.78:0/4071348369 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559c004540 con 0x7f55ac06ab40 2023-10-14T11:58:24.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.502+0000 7f5593fff700 1 -- 172.21.15.78:0/4071348369 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f559c0036a0 con 0x7f55ac06ab40 2023-10-14T11:58:24.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.502+0000 7f55b2002700 1 -- 172.21.15.78:0/4071348369 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55ac1ede40 con 0x7f55ac06ab40 2023-10-14T11:58:24.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.502+0000 7f5593fff700 1 -- 172.21.15.78:0/4071348369 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559c003a50 con 0x7f55ac06ab40 2023-10-14T11:58:24.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.502+0000 7f55b2002700 1 -- 172.21.15.78:0/4071348369 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55ac137c70 con 0x7f55ac06ab40 2023-10-14T11:58:24.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.503+0000 7f55b2002700 1 -- 172.21.15.78:0/4071348369 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55ac153f50 con 0x7f55ac06ab40 2023-10-14T11:58:24.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.504+0000 7f5593fff700 1 -- 172.21.15.78:0/4071348369 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f559c021070 con 0x7f55ac06ab40 2023-10-14T11:58:24.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.504+0000 7f5593fff700 1 --2- 172.21.15.78:0/4071348369 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f559806af00 0x7f559806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:24.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.504+0000 7f5593fff700 1 -- 172.21.15.78:0/4071348369 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f559c08ac00 con 0x7f55ac06ab40 2023-10-14T11:58:24.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.504+0000 7f55aaffd700 1 --2- 172.21.15.78:0/4071348369 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f559806af00 0x7f559806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:24.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.504+0000 7f55aaffd700 1 --2- 172.21.15.78:0/4071348369 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f559806af00 0x7f559806d3c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f55a0006fd0 tx=0x7f55a0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:24.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.508+0000 7f5593fff700 1 -- 172.21.15.78:0/4071348369 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f559c091080 con 0x7f55ac06ab40 2023-10-14T11:58:24.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.741+0000 7f55b2002700 1 -- 172.21.15.78:0/4071348369 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f55ac074b80 con 0x7f55ac06ab40 2023-10-14T11:58:24.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.741+0000 7f5593fff700 1 -- 172.21.15.78:0/4071348369 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f559c015020 con 0x7f55ac06ab40 2023-10-14T11:58:24.743 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:24.744 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-14T11:58:24.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.743+0000 7f55b2002700 1 -- 172.21.15.78:0/4071348369 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f559806af00 msgr2=0x7f559806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:24.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.743+0000 7f55b2002700 1 --2- 172.21.15.78:0/4071348369 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f559806af00 0x7f559806d3c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f55a0006fd0 tx=0x7f55a0008040 comp rx=0 tx=0).stop 2023-10-14T11:58:24.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.743+0000 7f55b2002700 1 -- 172.21.15.78:0/4071348369 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55ac06ab40 msgr2=0x7f55ac13b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:24.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.743+0000 7f55b2002700 1 --2- 172.21.15.78:0/4071348369 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55ac06ab40 0x7f55ac13b6c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f559c000c00 tx=0x7f559c004750 comp rx=0 tx=0).stop 2023-10-14T11:58:24.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.743+0000 7f55b2002700 1 -- 172.21.15.78:0/4071348369 shutdown_connections 2023-10-14T11:58:24.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.743+0000 7f55b2002700 1 --2- 172.21.15.78:0/4071348369 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f559806af00 0x7f559806d3c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:24.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.743+0000 7f55b2002700 1 --2- 172.21.15.78:0/4071348369 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f55ac1554e0 0x7f55ac1348a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:24.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.743+0000 7f55b2002700 1 --2- 172.21.15.78:0/4071348369 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55ac06ab40 0x7f55ac13b6c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:24.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.743+0000 7f55b2002700 1 --2- 172.21.15.78:0/4071348369 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f55ac0c0350 0x7f55ac134360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:24.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.743+0000 7f55b2002700 1 -- 172.21.15.78:0/4071348369 >> 172.21.15.78:0/4071348369 conn(0x7f55ac0c3ac0 msgr2=0x7f55ac0c68a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:24.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.743+0000 7f55b2002700 1 -- 172.21.15.78:0/4071348369 shutdown_connections 2023-10-14T11:58:24.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.743+0000 7f55b2002700 1 -- 172.21.15.78:0/4071348369 wait complete. 2023-10-14T11:58:24.757 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-14T11:58:24.758 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T11:58:24.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.875+0000 7fd0cfd20700 1 Processor -- start 2023-10-14T11:58:24.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.875+0000 7fd0cfd20700 1 -- start start 2023-10-14T11:58:24.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.876+0000 7fd0cfd20700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd0c806ab40 0x7fd0c806f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:24.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.876+0000 7fd0cfd20700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd0c81554e0 0x7fd0c806af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:24.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.876+0000 7fd0cfd20700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0c80c49f0 con 0x7fd0c80c0350 2023-10-14T11:58:24.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.876+0000 7fd0cfd20700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0c80726a0 con 0x7fd0c806ab40 2023-10-14T11:58:24.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.876+0000 7fd0cfd20700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0c8072980 con 0x7fd0c81554e0 2023-10-14T11:58:24.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.876+0000 7fd0cdabc700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd0c806ab40 0x7fd0c806f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:24.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.876+0000 7fd0ce2bd700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd0c80c0350 0x7fd0c813d7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49836/0 (socket says 172.21.15.78:49836) 2023-10-14T11:58:24.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.876+0000 7fd0ce2bd700 1 -- 172.21.15.78:0/3278651951 learned_addr learned my addr 172.21.15.78:0/3278651951 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:24.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.877+0000 7fd0cd2bb700 1 --2- 172.21.15.78:0/3278651951 >> v2:172.21.15.195:3300/0 conn(0x7fd0c81554e0 0x7fd0c806af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:24.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.877+0000 7fd0cdabc700 1 -- 172.21.15.78:0/3278651951 >> v2:172.21.15.195:3300/0 conn(0x7fd0c81554e0 msgr2=0x7fd0c806af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:24.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.877+0000 7fd0cdabc700 1 --2- 172.21.15.78:0/3278651951 >> v2:172.21.15.195:3300/0 conn(0x7fd0c81554e0 0x7fd0c806af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:24.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.877+0000 7fd0cdabc700 1 -- 172.21.15.78:0/3278651951 >> v1:172.21.15.78:6789/0 conn(0x7fd0c80c0350 legacy=0x7fd0c813d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:24.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.877+0000 7fd0cdabc700 1 -- 172.21.15.78:0/3278651951 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c8154a00 con 0x7fd0c806ab40 2023-10-14T11:58:24.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.877+0000 7fd0ccaba700 1 -- 172.21.15.78:0/3278651951 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 159139548 0 0) 0x7fd0c80c49f0 con 0x7fd0c80c0350 2023-10-14T11:58:24.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.877+0000 7fd0cdabc700 1 --2- 172.21.15.78:0/3278651951 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0c806ab40 0x7fd0c806f1e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd0c806ff00 tx=0x7fd0b8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eea3e762daece3f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:24.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.877+0000 7fd0ccaba700 1 -- 172.21.15.78:0/3278651951 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b8011830 con 0x7fd0c806ab40 2023-10-14T11:58:24.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.877+0000 7fd0ccaba700 1 -- 172.21.15.78:0/3278651951 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0b8011990 con 0x7fd0c806ab40 2023-10-14T11:58:24.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.877+0000 7fd0ccaba700 1 -- 172.21.15.78:0/3278651951 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b8011d20 con 0x7fd0c806ab40 2023-10-14T11:58:24.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.877+0000 7fd0cfd20700 1 -- 172.21.15.78:0/3278651951 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0c806ab40 msgr2=0x7fd0c806f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:24.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.877+0000 7fd0cfd20700 1 --2- 172.21.15.78:0/3278651951 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0c806ab40 0x7fd0c806f1e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd0c806ff00 tx=0x7fd0b8009b20 comp rx=0 tx=0).stop 2023-10-14T11:58:24.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.878+0000 7fd0cfd20700 1 -- 172.21.15.78:0/3278651951 shutdown_connections 2023-10-14T11:58:24.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.878+0000 7fd0cfd20700 1 --2- 172.21.15.78:0/3278651951 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0c806ab40 0x7fd0c806f1e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:24.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.878+0000 7fd0cfd20700 1 --2- 172.21.15.78:0/3278651951 >> v2:172.21.15.195:3300/0 conn(0x7fd0c81554e0 0x7fd0c806af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:24.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.878+0000 7fd0cfd20700 1 -- 172.21.15.78:0/3278651951 >> 172.21.15.78:0/3278651951 conn(0x7fd0c80c3ac0 msgr2=0x7fd0c80c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:24.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.878+0000 7fd0cfd20700 1 -- 172.21.15.78:0/3278651951 shutdown_connections 2023-10-14T11:58:24.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.878+0000 7fd0cfd20700 1 -- 172.21.15.78:0/3278651951 wait complete. 2023-10-14T11:58:24.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.879+0000 7fd0cfd20700 1 Processor -- start 2023-10-14T11:58:24.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.879+0000 7fd0cfd20700 1 -- start start 2023-10-14T11:58:24.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.879+0000 7fd0cfd20700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd0c806ab40 0x7fd0c8134de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:24.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.879+0000 7fd0cfd20700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0c80c0350 0x7fd0c80c5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:24.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.879+0000 7fd0cfd20700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd0c81554e0 0x7fd0c81e9410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:24.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.879+0000 7fd0cfd20700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c8156320 con 0x7fd0c80c0350 2023-10-14T11:58:24.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.879+0000 7fd0cfd20700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c8150cc0 con 0x7fd0c806ab40 2023-10-14T11:58:24.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.879+0000 7fd0cfd20700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c81564f0 con 0x7fd0c81554e0 2023-10-14T11:58:24.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.880+0000 7fd0cd2bb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0c80c0350 0x7fd0c80c5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:24.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.880+0000 7fd0cd2bb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0c80c0350 0x7fd0c80c5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44964/0 (socket says 172.21.15.78:44964) 2023-10-14T11:58:24.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.880+0000 7fd0cd2bb700 1 -- 172.21.15.78:0/621736565 learned_addr learned my addr 172.21.15.78:0/621736565 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:24.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.880+0000 7fd0cdabc700 1 --2- 172.21.15.78:0/621736565 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd0c806ab40 0x7fd0c8134de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:24.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.880+0000 7fd0ce2bd700 1 --2- 172.21.15.78:0/621736565 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd0c81554e0 0x7fd0c81e9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:24.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.880+0000 7fd0cd2bb700 1 -- 172.21.15.78:0/621736565 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd0c81554e0 msgr2=0x7fd0c81e9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:24.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.880+0000 7fd0cd2bb700 1 --2- 172.21.15.78:0/621736565 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd0c81554e0 0x7fd0c81e9410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:24.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.880+0000 7fd0cd2bb700 1 -- 172.21.15.78:0/621736565 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd0c806ab40 msgr2=0x7fd0c8134de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:24.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.880+0000 7fd0cd2bb700 1 --2- 172.21.15.78:0/621736565 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd0c806ab40 0x7fd0c8134de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:24.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.880+0000 7fd0cd2bb700 1 -- 172.21.15.78:0/621736565 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b80094a0 con 0x7fd0c80c0350 2023-10-14T11:58:24.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.880+0000 7fd0cd2bb700 1 --2- 172.21.15.78:0/621736565 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0c80c0350 0x7fd0c80c5e90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd0c400c7f0 tx=0x7fd0c400cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:24.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.880+0000 7fd0be7fc700 1 -- 172.21.15.78:0/621736565 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c4011800 con 0x7fd0c80c0350 2023-10-14T11:58:24.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.880+0000 7fd0be7fc700 1 -- 172.21.15.78:0/621736565 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0c4011960 con 0x7fd0c80c0350 2023-10-14T11:58:24.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.880+0000 7fd0be7fc700 1 -- 172.21.15.78:0/621736565 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c4011cd0 con 0x7fd0c80c0350 2023-10-14T11:58:24.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.880+0000 7fd0cfd20700 1 -- 172.21.15.78:0/621736565 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0c81567b0 con 0x7fd0c80c0350 2023-10-14T11:58:24.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.880+0000 7fd0cfd20700 1 -- 172.21.15.78:0/621736565 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0c8155c80 con 0x7fd0c80c0350 2023-10-14T11:58:24.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.881+0000 7fd0cfd20700 1 -- 172.21.15.78:0/621736565 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0c8150e30 con 0x7fd0c80c0350 2023-10-14T11:58:24.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.881+0000 7fd0be7fc700 1 -- 172.21.15.78:0/621736565 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd0c4010350 con 0x7fd0c80c0350 2023-10-14T11:58:24.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.882+0000 7fd0be7fc700 1 --2- 172.21.15.78:0/621736565 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd0ac06af00 0x7fd0ac06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:24.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.882+0000 7fd0be7fc700 1 -- 172.21.15.78:0/621736565 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd0c40890f0 con 0x7fd0c80c0350 2023-10-14T11:58:24.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.882+0000 7fd0cdabc700 1 --2- 172.21.15.78:0/621736565 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd0ac06af00 0x7fd0ac06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:24.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.882+0000 7fd0cdabc700 1 --2- 172.21.15.78:0/621736565 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd0ac06af00 0x7fd0ac06d3c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd0b80090b0 tx=0x7fd0b8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:24.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:24.886+0000 7fd0be7fc700 1 -- 172.21.15.78:0/621736565 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd0c40541b0 con 0x7fd0c80c0350 2023-10-14T11:58:25.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.078+0000 7fd0cfd20700 1 -- 172.21.15.78:0/621736565 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd0c8074b80 con 0x7fd0c80c0350 2023-10-14T11:58:25.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.079+0000 7fd0be7fc700 1 -- 172.21.15.78:0/621736565 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16481 (secure 0 0 0) 0x7fd0c40577d0 con 0x7fd0c80c0350 2023-10-14T11:58:25.081 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:25.082 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":21,"fsid":"4c5dc134-3092-406a-a1e1-13e6d9e926e8","created":"2023-10-14T11:57:57.363750+0000","modified":"2023-10-14T11:58:16.139413+0000","last_up_change":"2023-10-14T11:58:06.982472+0000","last_in_change":"2023-10-14T11:57:59.821373+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T11:58:09.001200+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T11:58:12.778573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T11:58:13.490286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"625a5fc0-9474-4eae-b80e-14c27a74ad19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6801","nonce":3884969780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6803","nonce":3884969780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6807","nonce":3884969780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6805","nonce":3884969780}]},"public_addr":"172.21.15.78:6801/3884969780","cluster_addr":"172.21.15.78:6803/3884969780","heartbeat_back_addr":"172.21.15.78:6807/3884969780","heartbeat_front_addr":"172.21.15.78:6805/3884969780","state":["exists","up"]},{"osd":1,"uuid":"49d63db1-2981-49ab-9c40-c50fc744b982","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6809","nonce":2176928406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6811","nonce":2176928406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6815","nonce":2176928406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6813","nonce":2176928406}]},"public_addr":"172.21.15.78:6809/2176928406","cluster_addr":"172.21.15.78:6811/2176928406","heartbeat_back_addr":"172.21.15.78:6815/2176928406","heartbeat_front_addr":"172.21.15.78:6813/2176928406","state":["exists","up"]},{"osd":2,"uuid":"1d7d0b29-cbd1-4717-9fe6-765a8d4998b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6825","nonce":3647044577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6827","nonce":3647044577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6831","nonce":3647044577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6829","nonce":3647044577}]},"public_addr":"172.21.15.78:6825/3647044577","cluster_addr":"172.21.15.78:6827/3647044577","heartbeat_back_addr":"172.21.15.78:6831/3647044577","heartbeat_front_addr":"172.21.15.78:6829/3647044577","state":["exists","up"]},{"osd":3,"uuid":"2f75ded1-e9e2-40e7-99ae-45e1e425e1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6817","nonce":3584112288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6819","nonce":3584112288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6823","nonce":3584112288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6821","nonce":3584112288}]},"public_addr":"172.21.15.78:6817/3584112288","cluster_addr":"172.21.15.78:6819/3584112288","heartbeat_back_addr":"172.21.15.78:6823/3584112288","heartbeat_front_addr":"172.21.15.78:6821/3584112288","state":["exists","up"]},{"osd":4,"uuid":"5130b432-8358-46cb-8a31-3064c976f6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6801","nonce":3732756301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6803","nonce":3732756301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6807","nonce":3732756301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6805","nonce":3732756301}]},"public_addr":"172.21.15.195:6801/3732756301","cluster_addr":"172.21.15.195:6803/3732756301","heartbeat_back_addr":"172.21.15.195:6807/3732756301","heartbeat_front_addr":"172.21.15.195:6805/3732756301","state":["exists","up"]},{"osd":5,"uuid":"60e234a9-a506-4e40-9c19-49a58f3cf28c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6825","nonce":2857706411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6827","nonce":2857706411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6831","nonce":2857706411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6829","nonce":2857706411}]},"public_addr":"172.21.15.195:6825/2857706411","cluster_addr":"172.21.15.195:6827/2857706411","heartbeat_back_addr":"172.21.15.195:6831/2857706411","heartbeat_front_addr":"172.21.15.195:6829/2857706411","state":["exists","up"]},{"osd":6,"uuid":"df7d2dd6-26b6-4eb1-96e5-94d1f3955b33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6810","nonce":72554374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6813","nonce":72554374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6820","nonce":72554374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6817","nonce":72554374}]},"public_addr":"172.21.15.195:6810/72554374","cluster_addr":"172.21.15.195:6813/72554374","heartbeat_back_addr":"172.21.15.195:6820/72554374","heartbeat_front_addr":"172.21.15.195:6817/72554374","state":["exists","up"]},{"osd":7,"uuid":"47a9d4de-f56d-4aa4-9900-c4a9cf9b2f97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6809","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6811","nonce":482472979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6816","nonce":482472979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6823","nonce":482472979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6821","nonce":482472979}]},"public_addr":"172.21.15.195:6811/482472979","cluster_addr":"172.21.15.195:6816/482472979","heartbeat_back_addr":"172.21.15.195:6823/482472979","heartbeat_front_addr":"172.21.15.195:6821/482472979","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.754003+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.005994+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.043603+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.024506+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.062585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.033767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.994825+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.038300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T11:58:25.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.081+0000 7fd0cfd20700 1 -- 172.21.15.78:0/621736565 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd0ac06af00 msgr2=0x7fd0ac06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:25.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.081+0000 7fd0cfd20700 1 --2- 172.21.15.78:0/621736565 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd0ac06af00 0x7fd0ac06d3c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd0b80090b0 tx=0x7fd0b8008030 comp rx=0 tx=0).stop 2023-10-14T11:58:25.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.081+0000 7fd0cfd20700 1 -- 172.21.15.78:0/621736565 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0c80c0350 msgr2=0x7fd0c80c5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:25.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.081+0000 7fd0cfd20700 1 --2- 172.21.15.78:0/621736565 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0c80c0350 0x7fd0c80c5e90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd0c400c7f0 tx=0x7fd0c400cb00 comp rx=0 tx=0).stop 2023-10-14T11:58:25.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.081+0000 7fd0cfd20700 1 -- 172.21.15.78:0/621736565 shutdown_connections 2023-10-14T11:58:25.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.081+0000 7fd0cfd20700 1 --2- 172.21.15.78:0/621736565 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd0c81554e0 0x7fd0c81e9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:25.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.081+0000 7fd0cfd20700 1 --2- 172.21.15.78:0/621736565 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd0ac06af00 0x7fd0ac06d3c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:25.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.081+0000 7fd0cfd20700 1 --2- 172.21.15.78:0/621736565 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd0c806ab40 0x7fd0c8134de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:25.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.081+0000 7fd0cfd20700 1 --2- 172.21.15.78:0/621736565 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0c80c0350 0x7fd0c80c5e90 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:25.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.081+0000 7fd0cfd20700 1 -- 172.21.15.78:0/621736565 >> 172.21.15.78:0/621736565 conn(0x7fd0c80c3ac0 msgr2=0x7fd0c80667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:25.086 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.082+0000 7fd0cfd20700 1 -- 172.21.15.78:0/621736565 shutdown_connections 2023-10-14T11:58:25.086 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.082+0000 7fd0cfd20700 1 -- 172.21.15.78:0/621736565 wait complete. 2023-10-14T11:58:25.095 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T11:58:09.001200+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T11:58:12.778573+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T11:58:13.490286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T11:58:25.096 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T11:58:25.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.218+0000 7f0541f7f700 1 Processor -- start 2023-10-14T11:58:25.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.218+0000 7f0541f7f700 1 -- start start 2023-10-14T11:58:25.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.219+0000 7f0541f7f700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f053c13dbe0 0x7f053c06a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:25.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.219+0000 7f0541f7f700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f053c06ae10 0x7f053c0c09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:25.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.219+0000 7f0541f7f700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f053c06ebe0 con 0x7f053c0c4540 2023-10-14T11:58:25.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.219+0000 7f0541f7f700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f053c06d330 con 0x7f053c13dbe0 2023-10-14T11:58:25.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.219+0000 7f0541f7f700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f053c1558c0 con 0x7f053c06ae10 2023-10-14T11:58:25.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.219+0000 7f053b7fe700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f053c13dbe0 0x7f053c06a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:25.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.219+0000 7f053b7fe700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f053c13dbe0 0x7f053c06a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44970/0 (socket says 172.21.15.78:44970) 2023-10-14T11:58:25.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.219+0000 7f053b7fe700 1 -- 172.21.15.78:0/3376483325 learned_addr learned my addr 172.21.15.78:0/3376483325 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:25.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.219+0000 7f053bfff700 1 --2- 172.21.15.78:0/3376483325 >> v2:172.21.15.195:3300/0 conn(0x7f053c06ae10 0x7f053c0c09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:25.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.219+0000 7f053b7fe700 1 -- 172.21.15.78:0/3376483325 >> v2:172.21.15.195:3300/0 conn(0x7f053c06ae10 msgr2=0x7f053c0c09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:25.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.219+0000 7f053b7fe700 1 --2- 172.21.15.78:0/3376483325 >> v2:172.21.15.195:3300/0 conn(0x7f053c06ae10 0x7f053c0c09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:25.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.219+0000 7f053b7fe700 1 -- 172.21.15.78:0/3376483325 >> v1:172.21.15.195:6789/0 conn(0x7f053c0c4540 legacy=0x7f053c0c02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:25.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.219+0000 7f053b7fe700 1 -- 172.21.15.78:0/3376483325 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053c154b00 con 0x7f053c13dbe0 2023-10-14T11:58:25.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.219+0000 7f053bfff700 1 --2- 172.21.15.78:0/3376483325 >> v2:172.21.15.195:3300/0 conn(0x7f053c06ae10 0x7f053c0c09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:25.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.219+0000 7f053b7fe700 1 --2- 172.21.15.78:0/3376483325 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f053c13dbe0 0x7f053c06a8d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f053c0c3a00 tx=0x7f052c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af55734ce8b67c4c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:25.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.219+0000 7f053a7fc700 1 -- 172.21.15.78:0/3376483325 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c004230 con 0x7f053c13dbe0 2023-10-14T11:58:25.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.219+0000 7f053a7fc700 1 -- 172.21.15.78:0/3376483325 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f052c004390 con 0x7f053c13dbe0 2023-10-14T11:58:25.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.219+0000 7f053a7fc700 1 -- 172.21.15.78:0/3376483325 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c0036a0 con 0x7f053c13dbe0 2023-10-14T11:58:25.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.220+0000 7f0541f7f700 1 -- 172.21.15.78:0/3376483325 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f053c13dbe0 msgr2=0x7f053c06a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:25.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.220+0000 7f0541f7f700 1 --2- 172.21.15.78:0/3376483325 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f053c13dbe0 0x7f053c06a8d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f053c0c3a00 tx=0x7f052c009bd0 comp rx=0 tx=0).stop 2023-10-14T11:58:25.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.220+0000 7f0541f7f700 1 -- 172.21.15.78:0/3376483325 shutdown_connections 2023-10-14T11:58:25.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.220+0000 7f0541f7f700 1 --2- 172.21.15.78:0/3376483325 >> v2:172.21.15.195:3300/0 conn(0x7f053c06ae10 0x7f053c0c09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:25.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.220+0000 7f0541f7f700 1 --2- 172.21.15.78:0/3376483325 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f053c13dbe0 0x7f053c06a8d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:25.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.220+0000 7f0541f7f700 1 -- 172.21.15.78:0/3376483325 >> 172.21.15.78:0/3376483325 conn(0x7f053c0c3a50 msgr2=0x7f053c0c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:25.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.220+0000 7f0541f7f700 1 -- 172.21.15.78:0/3376483325 shutdown_connections 2023-10-14T11:58:25.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.220+0000 7f0541f7f700 1 -- 172.21.15.78:0/3376483325 wait complete. 2023-10-14T11:58:25.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.221+0000 7f0541f7f700 1 Processor -- start 2023-10-14T11:58:25.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.221+0000 7f0541f7f700 1 -- start start 2023-10-14T11:58:25.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.222+0000 7f0541f7f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f053c06ae10 0x7f053c136c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:25.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.222+0000 7f0541f7f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f053c0c4540 0x7f053c1ed210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:25.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.222+0000 7f0541f7f700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f053c13dbe0 0x7f053c0ab810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:25.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.222+0000 7f0541f7f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053c158e50 con 0x7f053c06ae10 2023-10-14T11:58:25.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.222+0000 7f0541f7f700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053c1589c0 con 0x7f053c0c4540 2023-10-14T11:58:25.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.222+0000 7f0541f7f700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f053c158000 con 0x7f053c13dbe0 2023-10-14T11:58:25.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.222+0000 7f053b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f053c06ae10 0x7f053c136c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:25.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.222+0000 7f053b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f053c06ae10 0x7f053c136c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44976/0 (socket says 172.21.15.78:44976) 2023-10-14T11:58:25.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.222+0000 7f053b7fe700 1 -- 172.21.15.78:0/2874841219 learned_addr learned my addr 172.21.15.78:0/2874841219 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:25.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.222+0000 7f053affd700 1 --2- 172.21.15.78:0/2874841219 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f053c0c4540 0x7f053c1ed210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:25.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.222+0000 7f053b7fe700 1 -- 172.21.15.78:0/2874841219 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f053c13dbe0 msgr2=0x7f053c0ab810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:25.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.222+0000 7f053b7fe700 1 --2- 172.21.15.78:0/2874841219 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f053c13dbe0 0x7f053c0ab810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:25.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.222+0000 7f053b7fe700 1 -- 172.21.15.78:0/2874841219 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f053c0c4540 msgr2=0x7f053c1ed210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:25.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.222+0000 7f053b7fe700 1 --2- 172.21.15.78:0/2874841219 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f053c0c4540 0x7f053c1ed210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:25.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.222+0000 7f053b7fe700 1 -- 172.21.15.78:0/2874841219 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f052c009550 con 0x7f053c06ae10 2023-10-14T11:58:25.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.222+0000 7f053b7fe700 1 --2- 172.21.15.78:0/2874841219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f053c06ae10 0x7f053c136c10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f053c072fc0 tx=0x7f052c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:25.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.222+0000 7f0523fff700 1 -- 172.21.15.78:0/2874841219 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c011630 con 0x7f053c06ae10 2023-10-14T11:58:25.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.222+0000 7f0523fff700 1 -- 172.21.15.78:0/2874841219 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f052c011790 con 0x7f053c06ae10 2023-10-14T11:58:25.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.222+0000 7f0541f7f700 1 -- 172.21.15.78:0/2874841219 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f053c1edd00 con 0x7f053c06ae10 2023-10-14T11:58:25.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.222+0000 7f0523fff700 1 -- 172.21.15.78:0/2874841219 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c011b40 con 0x7f053c06ae10 2023-10-14T11:58:25.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.222+0000 7f0541f7f700 1 -- 172.21.15.78:0/2874841219 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f053c138480 con 0x7f053c06ae10 2023-10-14T11:58:25.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.223+0000 7f0541f7f700 1 -- 172.21.15.78:0/2874841219 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f053c074b80 con 0x7f053c06ae10 2023-10-14T11:58:25.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.224+0000 7f0523fff700 1 -- 172.21.15.78:0/2874841219 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f052c011ca0 con 0x7f053c06ae10 2023-10-14T11:58:25.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.224+0000 7f0523fff700 1 --2- 172.21.15.78:0/2874841219 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f052806af00 0x7f052806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:25.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.224+0000 7f0523fff700 1 -- 172.21.15.78:0/2874841219 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f052c08ac40 con 0x7f053c06ae10 2023-10-14T11:58:25.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.224+0000 7f053affd700 1 --2- 172.21.15.78:0/2874841219 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f052806af00 0x7f052806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:25.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.225+0000 7f053affd700 1 --2- 172.21.15.78:0/2874841219 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f052806af00 0x7f052806d3c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f053c0c6930 tx=0x7f0530009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:25.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.229+0000 7f0523fff700 1 -- 172.21.15.78:0/2874841219 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f052c055d80 con 0x7f053c06ae10 2023-10-14T11:58:25.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.417+0000 7f0541f7f700 1 -- 172.21.15.78:0/2874841219 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f053c074b80 con 0x7f053c06ae10 2023-10-14T11:58:25.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.418+0000 7f0523fff700 1 -- 172.21.15.78:0/2874841219 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f052c01f0a0 con 0x7f053c06ae10 2023-10-14T11:58:25.420 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-10-14T11:58:25.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.420+0000 7f0541f7f700 1 -- 172.21.15.78:0/2874841219 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f052806af00 msgr2=0x7f052806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:25.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.420+0000 7f0541f7f700 1 --2- 172.21.15.78:0/2874841219 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f052806af00 0x7f052806d3c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f053c0c6930 tx=0x7f0530009040 comp rx=0 tx=0).stop 2023-10-14T11:58:25.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.420+0000 7f0541f7f700 1 -- 172.21.15.78:0/2874841219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f053c06ae10 msgr2=0x7f053c136c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:25.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.420+0000 7f0541f7f700 1 --2- 172.21.15.78:0/2874841219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f053c06ae10 0x7f053c136c10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f053c072fc0 tx=0x7f052c004750 comp rx=0 tx=0).stop 2023-10-14T11:58:25.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.420+0000 7f0541f7f700 1 -- 172.21.15.78:0/2874841219 shutdown_connections 2023-10-14T11:58:25.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.420+0000 7f0541f7f700 1 --2- 172.21.15.78:0/2874841219 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f053c13dbe0 0x7f053c0ab810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:25.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.420+0000 7f0541f7f700 1 --2- 172.21.15.78:0/2874841219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f053c06ae10 0x7f053c136c10 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:25.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.420+0000 7f0541f7f700 1 --2- 172.21.15.78:0/2874841219 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f052806af00 0x7f052806d3c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:25.424 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.420+0000 7f0541f7f700 1 --2- 172.21.15.78:0/2874841219 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f053c0c4540 0x7f053c1ed210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:25.424 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.420+0000 7f0541f7f700 1 -- 172.21.15.78:0/2874841219 >> 172.21.15.78:0/2874841219 conn(0x7f053c0c3a50 msgr2=0x7f053c0c31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:25.424 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.420+0000 7f0541f7f700 1 -- 172.21.15.78:0/2874841219 shutdown_connections 2023-10-14T11:58:25.424 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.420+0000 7f0541f7f700 1 -- 172.21.15.78:0/2874841219 wait complete. 2023-10-14T11:58:25.434 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T11:58:25.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.555+0000 7fd1c084f700 1 Processor -- start 2023-10-14T11:58:25.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.555+0000 7fd1c084f700 1 -- start start 2023-10-14T11:58:25.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.556+0000 7fd1c084f700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd1bc0bd950 0x7fd1bc0c70b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:25.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.556+0000 7fd1c084f700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1bc0c4c40 con 0x7fd1bc0c6cd0 2023-10-14T11:58:25.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.556+0000 7fd1c084f700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1bc0c54e0 con 0x7fd1bc14ebd0 2023-10-14T11:58:25.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.556+0000 7fd1c084f700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc1556d0 con 0x7fd1bc0bd950 2023-10-14T11:58:25.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.556+0000 7fd1ba59c700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd1bc0c6cd0 0x7fd1bc1581d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49860/0 (socket says 172.21.15.78:49860) 2023-10-14T11:58:25.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.556+0000 7fd1ba59c700 1 -- 172.21.15.78:0/1229739391 learned_addr learned my addr 172.21.15.78:0/1229739391 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:25.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.556+0000 7fd1b959a700 1 -- 172.21.15.78:0/1229739391 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3253769480 0 0) 0x7fd1bc0c4c40 con 0x7fd1bc0c6cd0 2023-10-14T11:58:25.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.556+0000 7fd1b959a700 1 -- 172.21.15.78:0/1229739391 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1a40035b0 con 0x7fd1bc0c6cd0 2023-10-14T11:58:25.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.556+0000 7fd1bad9d700 1 --2- 172.21.15.78:0/1229739391 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd1bc0bd950 0x7fd1bc0c70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:25.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.556+0000 7fd1b959a700 1 -- 172.21.15.78:0/1229739391 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4223803951 0 0) 0x7fd1a40035b0 con 0x7fd1bc0c6cd0 2023-10-14T11:58:25.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.556+0000 7fd1b959a700 1 -- 172.21.15.78:0/1229739391 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd1bc0bd950 msgr2=0x7fd1bc0c70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:25.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.556+0000 7fd1b959a700 1 --2- 172.21.15.78:0/1229739391 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd1bc0bd950 0x7fd1bc0c70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:25.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.556+0000 7fd1b959a700 1 -- 172.21.15.78:0/1229739391 >> v1:172.21.15.195:6789/0 conn(0x7fd1bc14ebd0 legacy=0x7fd1bc1588e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:25.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.556+0000 7fd1b959a700 1 -- 172.21.15.78:0/1229739391 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1bc0c4e90 con 0x7fd1bc0c6cd0 2023-10-14T11:58:25.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.556+0000 7fd1b959a700 1 -- 172.21.15.78:0/1229739391 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441799917 0 0) 0x7fd1b0001670 con 0x7fd1bc0c6cd0 2023-10-14T11:58:25.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.557+0000 7fd1b959a700 1 -- 172.21.15.78:0/1229739391 >> v1:172.21.15.78:6789/0 conn(0x7fd1bc0c6cd0 legacy=0x7fd1bc1581d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:25.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.557+0000 7fd1b959a700 1 --2- 172.21.15.78:0/1229739391 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd1a4003a30 0x7fd1a4003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:25.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.557+0000 7fd1b959a700 1 --2- 172.21.15.78:0/1229739391 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1a4004850 0x7fd1a4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:25.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.557+0000 7fd1b959a700 1 --2- 172.21.15.78:0/1229739391 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd1bc0bd950 0x7fd1a400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:25.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.557+0000 7fd1b959a700 1 -- 172.21.15.78:0/1229739391 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc1556d0 con 0x7fd1a4004850 2023-10-14T11:58:25.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.557+0000 7fd1b959a700 1 -- 172.21.15.78:0/1229739391 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a4005570 con 0x7fd1a4003a30 2023-10-14T11:58:25.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.557+0000 7fd1b959a700 1 -- 172.21.15.78:0/1229739391 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a400bf50 con 0x7fd1bc0bd950 2023-10-14T11:58:25.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.557+0000 7fd1b9d9b700 1 --2- 172.21.15.78:0/1229739391 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1a4004850 0x7fd1a4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:25.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.557+0000 7fd1ba59c700 1 --2- 172.21.15.78:0/1229739391 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd1a4003a30 0x7fd1a4003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:25.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.557+0000 7fd1bad9d700 1 --2- 172.21.15.78:0/1229739391 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd1bc0bd950 0x7fd1a400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:25.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.557+0000 7fd1b9d9b700 1 -- 172.21.15.78:0/1229739391 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd1bc0bd950 msgr2=0x7fd1a400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:25.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.557+0000 7fd1b9d9b700 1 --2- 172.21.15.78:0/1229739391 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd1bc0bd950 0x7fd1a400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:25.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.557+0000 7fd1b9d9b700 1 -- 172.21.15.78:0/1229739391 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd1a4003a30 msgr2=0x7fd1a4003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:25.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.557+0000 7fd1b9d9b700 1 --2- 172.21.15.78:0/1229739391 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd1a4003a30 0x7fd1a4003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:25.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.557+0000 7fd1b9d9b700 1 -- 172.21.15.78:0/1229739391 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a40035b0 con 0x7fd1a4004850 2023-10-14T11:58:25.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.557+0000 7fd1ba59c700 1 --2- 172.21.15.78:0/1229739391 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd1a4003a30 0x7fd1a4003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:25.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.557+0000 7fd1b9d9b700 1 --2- 172.21.15.78:0/1229739391 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1a4004850 0x7fd1a4004cd0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc0acdc0 tx=0x7fd1a4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad8ec875bbdcd1ae server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:25.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.557+0000 7fd1b959a700 1 -- 172.21.15.78:0/1229739391 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1ac0012f0 con 0x7fd1a4004850 2023-10-14T11:58:25.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.557+0000 7fd1b959a700 1 -- 172.21.15.78:0/1229739391 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1ac0052f0 con 0x7fd1a4004850 2023-10-14T11:58:25.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.557+0000 7fd1b959a700 1 -- 172.21.15.78:0/1229739391 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1ac006860 con 0x7fd1a4004850 2023-10-14T11:58:25.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.558+0000 7fd1c084f700 1 -- 172.21.15.78:0/1229739391 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1a4004850 msgr2=0x7fd1a4004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:25.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.558+0000 7fd1c084f700 1 --2- 172.21.15.78:0/1229739391 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1a4004850 0x7fd1a4004cd0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc0acdc0 tx=0x7fd1a4005880 comp rx=0 tx=0).stop 2023-10-14T11:58:25.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.558+0000 7fd1bad9d700 1 -- 172.21.15.78:0/1229739391 reap_dead start 2023-10-14T11:58:25.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.558+0000 7fd1c084f700 1 -- 172.21.15.78:0/1229739391 shutdown_connections 2023-10-14T11:58:25.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.558+0000 7fd1c084f700 1 -- 172.21.15.78:0/1229739391 >> 172.21.15.78:0/1229739391 conn(0x7fd1bc0abdf0 msgr2=0x7fd1bc0ac1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:25.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.558+0000 7fd1c084f700 1 -- 172.21.15.78:0/1229739391 shutdown_connections 2023-10-14T11:58:25.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.558+0000 7fd1c084f700 1 -- 172.21.15.78:0/1229739391 wait complete. 2023-10-14T11:58:25.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.559+0000 7fd1c084f700 1 Processor -- start 2023-10-14T11:58:25.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.559+0000 7fd1c084f700 1 -- start start 2023-10-14T11:58:25.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.560+0000 7fd1c084f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd1a4004850 0x7fd1bc1ed040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:25.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.560+0000 7fd1c084f700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd1bc151930 0x7fd1bc0bf4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:25.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.560+0000 7fd1c084f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1bc0bfa30 0x7fd1bc14f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:25.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.560+0000 7fd1c084f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc14b450 con 0x7fd1bc0bfa30 2023-10-14T11:58:25.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.560+0000 7fd1c084f700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc1514e0 con 0x7fd1a4004850 2023-10-14T11:58:25.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.560+0000 7fd1c084f700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc0c0070 con 0x7fd1bc151930 2023-10-14T11:58:25.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.560+0000 7fd1bad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1bc0bfa30 0x7fd1bc14f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:25.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.560+0000 7fd1bad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1bc0bfa30 0x7fd1bc14f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45000/0 (socket says 172.21.15.78:45000) 2023-10-14T11:58:25.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.560+0000 7fd1bad9d700 1 -- 172.21.15.78:0/373671680 learned_addr learned my addr 172.21.15.78:0/373671680 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:25.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.560+0000 7fd1b9d9b700 1 --2- 172.21.15.78:0/373671680 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd1bc151930 0x7fd1bc0bf4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:25.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.560+0000 7fd1ba59c700 1 --2- 172.21.15.78:0/373671680 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd1a4004850 0x7fd1bc1ed040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:25.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.560+0000 7fd1bad9d700 1 -- 172.21.15.78:0/373671680 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd1bc151930 msgr2=0x7fd1bc0bf4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:25.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.560+0000 7fd1bad9d700 1 --2- 172.21.15.78:0/373671680 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd1bc151930 0x7fd1bc0bf4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:25.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.560+0000 7fd1bad9d700 1 -- 172.21.15.78:0/373671680 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd1a4004850 msgr2=0x7fd1bc1ed040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:25.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.560+0000 7fd1bad9d700 1 --2- 172.21.15.78:0/373671680 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd1a4004850 0x7fd1bc1ed040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:25.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.560+0000 7fd1bad9d700 1 -- 172.21.15.78:0/373671680 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1bc14bf70 con 0x7fd1bc0bfa30 2023-10-14T11:58:25.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.560+0000 7fd1bad9d700 1 --2- 172.21.15.78:0/373671680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1bc0bfa30 0x7fd1bc14f9d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc06b090 tx=0x7fd1b40048c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:25.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.560+0000 7fd1aaffd700 1 -- 172.21.15.78:0/373671680 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b40078a0 con 0x7fd1bc0bfa30 2023-10-14T11:58:25.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.561+0000 7fd1aaffd700 1 -- 172.21.15.78:0/373671680 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1b4007a00 con 0x7fd1bc0bfa30 2023-10-14T11:58:25.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.561+0000 7fd1c084f700 1 -- 172.21.15.78:0/373671680 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1bc14bcd0 con 0x7fd1bc0bfa30 2023-10-14T11:58:25.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.561+0000 7fd1aaffd700 1 -- 172.21.15.78:0/373671680 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b4007d90 con 0x7fd1bc0bfa30 2023-10-14T11:58:25.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.561+0000 7fd1c084f700 1 -- 172.21.15.78:0/373671680 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1bc1edc60 con 0x7fd1bc0bfa30 2023-10-14T11:58:25.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.561+0000 7fd1c084f700 1 -- 172.21.15.78:0/373671680 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1bc151650 con 0x7fd1bc0bfa30 2023-10-14T11:58:25.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.562+0000 7fd1aaffd700 1 -- 172.21.15.78:0/373671680 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd1b4028430 con 0x7fd1bc0bfa30 2023-10-14T11:58:25.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.563+0000 7fd1aaffd700 1 --2- 172.21.15.78:0/373671680 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd19c06af00 0x7fd19c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:25.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.563+0000 7fd1aaffd700 1 -- 172.21.15.78:0/373671680 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd1b4089fc0 con 0x7fd1bc0bfa30 2023-10-14T11:58:25.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.563+0000 7fd1ba59c700 1 --2- 172.21.15.78:0/373671680 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd19c06af00 0x7fd19c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:25.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.563+0000 7fd1ba59c700 1 --2- 172.21.15.78:0/373671680 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd19c06af00 0x7fd19c06d3c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0005ba0 tx=0x7fd1b0007040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:25.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.567+0000 7fd1aaffd700 1 -- 172.21.15.78:0/373671680 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd1b40551e0 con 0x7fd1bc0bfa30 2023-10-14T11:58:25.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.755+0000 7fd1c084f700 1 -- 172.21.15.78:0/373671680 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd1bc14bf70 con 0x7fd1bc0bfa30 2023-10-14T11:58:25.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.755+0000 7fd1aaffd700 1 -- 172.21.15.78:0/373671680 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd1bc14bf70 con 0x7fd1bc0bfa30 2023-10-14T11:58:25.758 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-10-14T11:58:25.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.757+0000 7fd1c084f700 1 -- 172.21.15.78:0/373671680 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd19c06af00 msgr2=0x7fd19c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:25.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.757+0000 7fd1c084f700 1 --2- 172.21.15.78:0/373671680 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd19c06af00 0x7fd19c06d3c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0005ba0 tx=0x7fd1b0007040 comp rx=0 tx=0).stop 2023-10-14T11:58:25.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.757+0000 7fd1c084f700 1 -- 172.21.15.78:0/373671680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1bc0bfa30 msgr2=0x7fd1bc14f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:25.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.757+0000 7fd1c084f700 1 --2- 172.21.15.78:0/373671680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1bc0bfa30 0x7fd1bc14f9d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc06b090 tx=0x7fd1b40048c0 comp rx=0 tx=0).stop 2023-10-14T11:58:25.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.757+0000 7fd1c084f700 1 -- 172.21.15.78:0/373671680 shutdown_connections 2023-10-14T11:58:25.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.757+0000 7fd1c084f700 1 --2- 172.21.15.78:0/373671680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1bc0bfa30 0x7fd1bc14f9d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:25.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.757+0000 7fd1c084f700 1 --2- 172.21.15.78:0/373671680 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd19c06af00 0x7fd19c06d3c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:25.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.757+0000 7fd1c084f700 1 --2- 172.21.15.78:0/373671680 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd1a4004850 0x7fd1bc1ed040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:25.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.757+0000 7fd1c084f700 1 --2- 172.21.15.78:0/373671680 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd1bc151930 0x7fd1bc0bf4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:25.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.757+0000 7fd1c084f700 1 -- 172.21.15.78:0/373671680 >> 172.21.15.78:0/373671680 conn(0x7fd1bc0abdf0 msgr2=0x7fd1bc14d390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:25.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.757+0000 7fd1c084f700 1 -- 172.21.15.78:0/373671680 shutdown_connections 2023-10-14T11:58:25.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.757+0000 7fd1c084f700 1 -- 172.21.15.78:0/373671680 wait complete. 2023-10-14T11:58:25.771 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T11:58:25.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.891+0000 7f3d57717700 1 Processor -- start 2023-10-14T11:58:25.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.892+0000 7f3d57717700 1 -- start start 2023-10-14T11:58:25.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.892+0000 7f3d57717700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3d50155070 0x7f3d5015e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:25.898 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.892+0000 7f3d57717700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d501537b0 con 0x7f3d500a70c0 2023-10-14T11:58:25.898 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.892+0000 7f3d57717700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d500c7c50 con 0x7f3d501533d0 2023-10-14T11:58:25.898 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.892+0000 7f3d57717700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d500c6fb0 con 0x7f3d50155070 2023-10-14T11:58:25.898 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.893+0000 7f3d54cb2700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f3d500a70c0 0x7f3d50154960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49882/0 (socket says 172.21.15.78:49882) 2023-10-14T11:58:25.899 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.893+0000 7f3d54cb2700 1 -- 172.21.15.78:0/3079695 learned_addr learned my addr 172.21.15.78:0/3079695 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:25.899 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.893+0000 7f3d47fff700 1 -- 172.21.15.78:0/3079695 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4092305630 0 0) 0x7f3d501537b0 con 0x7f3d500a70c0 2023-10-14T11:58:25.899 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.893+0000 7f3d55cb4700 1 --2- 172.21.15.78:0/3079695 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3d50155070 0x7f3d5015e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:25.899 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.893+0000 7f3d47fff700 1 -- 172.21.15.78:0/3079695 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d3c0035b0 con 0x7f3d500a70c0 2023-10-14T11:58:25.899 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.893+0000 7f3d47fff700 1 -- 172.21.15.78:0/3079695 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1944574782 0 0) 0x7f3d3c0035b0 con 0x7f3d500a70c0 2023-10-14T11:58:25.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.893+0000 7f3d47fff700 1 -- 172.21.15.78:0/3079695 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3d50155070 msgr2=0x7f3d5015e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:25.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.893+0000 7f3d47fff700 1 --2- 172.21.15.78:0/3079695 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3d50155070 0x7f3d5015e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:25.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.893+0000 7f3d47fff700 1 -- 172.21.15.78:0/3079695 >> v1:172.21.15.195:6789/0 conn(0x7f3d501533d0 legacy=0x7f3d500c8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:25.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.893+0000 7f3d47fff700 1 -- 172.21.15.78:0/3079695 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d50153e80 con 0x7f3d500a70c0 2023-10-14T11:58:25.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.893+0000 7f3d47fff700 1 -- 172.21.15.78:0/3079695 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441799917 0 0) 0x7f3d4c001670 con 0x7f3d500a70c0 2023-10-14T11:58:25.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.893+0000 7f3d47fff700 1 -- 172.21.15.78:0/3079695 >> v1:172.21.15.78:6789/0 conn(0x7f3d500a70c0 legacy=0x7f3d50154960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:25.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.893+0000 7f3d47fff700 1 --2- 172.21.15.78:0/3079695 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3d50155070 0x7f3d3c003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:25.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.893+0000 7f3d47fff700 1 --2- 172.21.15.78:0/3079695 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d3c003fb0 0x7f3d3c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:25.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.893+0000 7f3d47fff700 1 --2- 172.21.15.78:0/3079695 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3d3c004d90 0x7f3d3c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:25.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.893+0000 7f3d47fff700 1 -- 172.21.15.78:0/3079695 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d500c6fb0 con 0x7f3d3c003fb0 2023-10-14T11:58:25.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.893+0000 7f3d47fff700 1 -- 172.21.15.78:0/3079695 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c00bba0 con 0x7f3d3c004d90 2023-10-14T11:58:25.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.893+0000 7f3d47fff700 1 -- 172.21.15.78:0/3079695 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c00bd60 con 0x7f3d50155070 2023-10-14T11:58:25.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.893+0000 7f3d54cb2700 1 --2- 172.21.15.78:0/3079695 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d3c003fb0 0x7f3d3c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:25.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.893+0000 7f3d554b3700 1 --2- 172.21.15.78:0/3079695 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3d50155070 0x7f3d3c003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:25.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.893+0000 7f3d55cb4700 1 --2- 172.21.15.78:0/3079695 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3d3c004d90 0x7f3d3c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:25.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.893+0000 7f3d54cb2700 1 -- 172.21.15.78:0/3079695 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3d50155070 msgr2=0x7f3d3c003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:25.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.893+0000 7f3d54cb2700 1 --2- 172.21.15.78:0/3079695 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3d50155070 0x7f3d3c003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:25.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.893+0000 7f3d54cb2700 1 -- 172.21.15.78:0/3079695 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3d3c004d90 msgr2=0x7f3d3c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:25.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.893+0000 7f3d54cb2700 1 --2- 172.21.15.78:0/3079695 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3d3c004d90 0x7f3d3c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:25.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.893+0000 7f3d54cb2700 1 -- 172.21.15.78:0/3079695 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d3c0035b0 con 0x7f3d3c003fb0 2023-10-14T11:58:25.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.894+0000 7f3d55cb4700 1 --2- 172.21.15.78:0/3079695 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3d3c004d90 0x7f3d3c005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:25.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.894+0000 7f3d54cb2700 1 --2- 172.21.15.78:0/3079695 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d3c003fb0 0x7f3d3c004850 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c00c2a0 tx=0x7f3d3c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6b65d41a27348a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:25.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.894+0000 7f3d47fff700 1 -- 172.21.15.78:0/3079695 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c007c10 con 0x7f3d3c003fb0 2023-10-14T11:58:25.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.894+0000 7f3d47fff700 1 -- 172.21.15.78:0/3079695 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d4c00cdc0 con 0x7f3d3c003fb0 2023-10-14T11:58:25.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.894+0000 7f3d47fff700 1 -- 172.21.15.78:0/3079695 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c005ba0 con 0x7f3d3c003fb0 2023-10-14T11:58:25.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.894+0000 7f3d57717700 1 -- 172.21.15.78:0/3079695 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d3c003fb0 msgr2=0x7f3d3c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:25.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.894+0000 7f3d57717700 1 --2- 172.21.15.78:0/3079695 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d3c003fb0 0x7f3d3c004850 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c00c2a0 tx=0x7f3d3c005880 comp rx=0 tx=0).stop 2023-10-14T11:58:25.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.894+0000 7f3d55cb4700 1 -- 172.21.15.78:0/3079695 reap_dead start 2023-10-14T11:58:25.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.894+0000 7f3d57717700 1 -- 172.21.15.78:0/3079695 shutdown_connections 2023-10-14T11:58:25.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.894+0000 7f3d57717700 1 -- 172.21.15.78:0/3079695 >> 172.21.15.78:0/3079695 conn(0x7f3d500737d0 msgr2=0x7f3d50073be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:25.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.894+0000 7f3d57717700 1 -- 172.21.15.78:0/3079695 shutdown_connections 2023-10-14T11:58:25.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.894+0000 7f3d57717700 1 -- 172.21.15.78:0/3079695 wait complete. 2023-10-14T11:58:25.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.896+0000 7f3d57717700 1 Processor -- start 2023-10-14T11:58:25.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.896+0000 7f3d57717700 1 -- start start 2023-10-14T11:58:25.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.896+0000 7f3d57717700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3d3c003fb0 0x7f3d501e8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:25.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.896+0000 7f3d57717700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3d501e76a0 0x7f3d500ad5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:25.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.896+0000 7f3d57717700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d500aa370 0x7f3d5014bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:25.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.896+0000 7f3d57717700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d500c7f30 con 0x7f3d500aa370 2023-10-14T11:58:25.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.896+0000 7f3d57717700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d500c7a70 con 0x7f3d501e76a0 2023-10-14T11:58:25.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.896+0000 7f3d57717700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d500ac2d0 con 0x7f3d3c003fb0 2023-10-14T11:58:25.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.896+0000 7f3d55cb4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d500aa370 0x7f3d5014bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:25.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.896+0000 7f3d55cb4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d500aa370 0x7f3d5014bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45020/0 (socket says 172.21.15.78:45020) 2023-10-14T11:58:25.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.896+0000 7f3d55cb4700 1 -- 172.21.15.78:0/905221899 learned_addr learned my addr 172.21.15.78:0/905221899 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:25.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.896+0000 7f3d554b3700 1 --2- 172.21.15.78:0/905221899 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3d3c003fb0 0x7f3d501e8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:25.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.896+0000 7f3d54cb2700 1 --2- 172.21.15.78:0/905221899 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3d501e76a0 0x7f3d500ad5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:25.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.897+0000 7f3d55cb4700 1 -- 172.21.15.78:0/905221899 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3d3c003fb0 msgr2=0x7f3d501e8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:25.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.897+0000 7f3d55cb4700 1 --2- 172.21.15.78:0/905221899 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3d3c003fb0 0x7f3d501e8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:25.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.897+0000 7f3d55cb4700 1 -- 172.21.15.78:0/905221899 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3d501e76a0 msgr2=0x7f3d500ad5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:25.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.897+0000 7f3d55cb4700 1 --2- 172.21.15.78:0/905221899 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3d501e76a0 0x7f3d500ad5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:25.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.897+0000 7f3d55cb4700 1 -- 172.21.15.78:0/905221899 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d501e9050 con 0x7f3d500aa370 2023-10-14T11:58:25.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.897+0000 7f3d54cb2700 1 --2- 172.21.15.78:0/905221899 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3d501e76a0 0x7f3d500ad5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:25.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.897+0000 7f3d55cb4700 1 --2- 172.21.15.78:0/905221899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d500aa370 0x7f3d5014bba0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3d500c6df0 tx=0x7f3d480104a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:25.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.897+0000 7f3d45ffb700 1 -- 172.21.15.78:0/905221899 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4800e050 con 0x7f3d500aa370 2023-10-14T11:58:25.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.897+0000 7f3d45ffb700 1 -- 172.21.15.78:0/905221899 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d4800f040 con 0x7f3d500aa370 2023-10-14T11:58:25.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.897+0000 7f3d45ffb700 1 -- 172.21.15.78:0/905221899 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d48010df0 con 0x7f3d500aa370 2023-10-14T11:58:25.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.897+0000 7f3d57717700 1 -- 172.21.15.78:0/905221899 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d500ac520 con 0x7f3d500aa370 2023-10-14T11:58:25.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.897+0000 7f3d57717700 1 -- 172.21.15.78:0/905221899 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d500acae0 con 0x7f3d500aa370 2023-10-14T11:58:25.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.898+0000 7f3d57717700 1 -- 172.21.15.78:0/905221899 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d50074b80 con 0x7f3d500aa370 2023-10-14T11:58:25.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.898+0000 7f3d45ffb700 1 -- 172.21.15.78:0/905221899 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3d48015440 con 0x7f3d500aa370 2023-10-14T11:58:25.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.899+0000 7f3d45ffb700 1 --2- 172.21.15.78:0/905221899 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f3d3406af00 0x7f3d3406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:25.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.899+0000 7f3d45ffb700 1 -- 172.21.15.78:0/905221899 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3d48089630 con 0x7f3d500aa370 2023-10-14T11:58:25.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.899+0000 7f3d554b3700 1 --2- 172.21.15.78:0/905221899 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f3d3406af00 0x7f3d3406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:25.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.899+0000 7f3d554b3700 1 --2- 172.21.15.78:0/905221899 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f3d3406af00 0x7f3d3406d3c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3d500c4430 tx=0x7f3d40009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:25.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:25.903+0000 7f3d45ffb700 1 -- 172.21.15.78:0/905221899 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3d48054770 con 0x7f3d500aa370 2023-10-14T11:58:26.097 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.096+0000 7f3d57717700 1 -- 172.21.15.78:0/905221899 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3d500c70f0 con 0x7f3d500aa370 2023-10-14T11:58:26.098 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.096+0000 7f3d45ffb700 1 -- 172.21.15.78:0/905221899 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3d48057d90 con 0x7f3d500aa370 2023-10-14T11:58:26.099 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2023-10-14T11:58:26.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.098+0000 7f3d57717700 1 -- 172.21.15.78:0/905221899 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f3d3406af00 msgr2=0x7f3d3406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:26.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.098+0000 7f3d57717700 1 --2- 172.21.15.78:0/905221899 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f3d3406af00 0x7f3d3406d3c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3d500c4430 tx=0x7f3d40009040 comp rx=0 tx=0).stop 2023-10-14T11:58:26.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.098+0000 7f3d57717700 1 -- 172.21.15.78:0/905221899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d500aa370 msgr2=0x7f3d5014bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:26.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.098+0000 7f3d57717700 1 --2- 172.21.15.78:0/905221899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d500aa370 0x7f3d5014bba0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3d500c6df0 tx=0x7f3d480104a0 comp rx=0 tx=0).stop 2023-10-14T11:58:26.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.098+0000 7f3d57717700 1 -- 172.21.15.78:0/905221899 shutdown_connections 2023-10-14T11:58:26.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.098+0000 7f3d57717700 1 --2- 172.21.15.78:0/905221899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d500aa370 0x7f3d5014bba0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:26.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.098+0000 7f3d57717700 1 --2- 172.21.15.78:0/905221899 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3d3c003fb0 0x7f3d501e8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:26.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.098+0000 7f3d57717700 1 --2- 172.21.15.78:0/905221899 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f3d3406af00 0x7f3d3406d3c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:26.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.098+0000 7f3d57717700 1 --2- 172.21.15.78:0/905221899 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3d501e76a0 0x7f3d500ad5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:26.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.098+0000 7f3d57717700 1 -- 172.21.15.78:0/905221899 >> 172.21.15.78:0/905221899 conn(0x7f3d500737d0 msgr2=0x7f3d500c31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:26.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.098+0000 7f3d57717700 1 -- 172.21.15.78:0/905221899 shutdown_connections 2023-10-14T11:58:26.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.098+0000 7f3d57717700 1 -- 172.21.15.78:0/905221899 wait complete. 2023-10-14T11:58:26.112 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T11:58:26.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.232+0000 7fe661eed700 1 Processor -- start 2023-10-14T11:58:26.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.232+0000 7fe661eed700 1 -- start start 2023-10-14T11:58:26.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.232+0000 7fe661eed700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe65c1554e0 0x7fe65c13d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:26.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.232+0000 7fe661eed700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe65c06b630 con 0x7fe65c06ab40 2023-10-14T11:58:26.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.232+0000 7fe661eed700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe65c0c0a60 con 0x7fe65c06f1e0 2023-10-14T11:58:26.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.232+0000 7fe661eed700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe65c155930 con 0x7fe65c1554e0 2023-10-14T11:58:26.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.233+0000 7fe65b7fe700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fe65c06ab40 0x7fe65c06af20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49900/0 (socket says 172.21.15.78:49900) 2023-10-14T11:58:26.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.233+0000 7fe65b7fe700 1 -- 172.21.15.78:0/1181257833 learned_addr learned my addr 172.21.15.78:0/1181257833 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:26.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.233+0000 7fe65bfff700 1 --2- 172.21.15.78:0/1181257833 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe65c1554e0 0x7fe65c13d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:26.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.233+0000 7fe65a7fc700 1 -- 172.21.15.78:0/1181257833 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3536686352 0 0) 0x7fe65c06b630 con 0x7fe65c06ab40 2023-10-14T11:58:26.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.233+0000 7fe65a7fc700 1 -- 172.21.15.78:0/1181257833 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6440035b0 con 0x7fe65c06ab40 2023-10-14T11:58:26.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.233+0000 7fe65a7fc700 1 -- 172.21.15.78:0/1181257833 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2871751737 0 0) 0x7fe6440035b0 con 0x7fe65c06ab40 2023-10-14T11:58:26.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.233+0000 7fe65a7fc700 1 -- 172.21.15.78:0/1181257833 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe65c1554e0 msgr2=0x7fe65c13d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:26.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.233+0000 7fe65a7fc700 1 --2- 172.21.15.78:0/1181257833 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe65c1554e0 0x7fe65c13d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:26.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.233+0000 7fe65a7fc700 1 -- 172.21.15.78:0/1181257833 >> v1:172.21.15.195:6789/0 conn(0x7fe65c06f1e0 legacy=0x7fe65c0c0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:26.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.233+0000 7fe65a7fc700 1 -- 172.21.15.78:0/1181257833 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe65c0c49f0 con 0x7fe65c06ab40 2023-10-14T11:58:26.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.233+0000 7fe65a7fc700 1 -- 172.21.15.78:0/1181257833 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441799917 0 0) 0x7fe64c001670 con 0x7fe65c06ab40 2023-10-14T11:58:26.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.233+0000 7fe65a7fc700 1 -- 172.21.15.78:0/1181257833 >> v1:172.21.15.78:6789/0 conn(0x7fe65c06ab40 legacy=0x7fe65c06af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:26.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.233+0000 7fe65a7fc700 1 --2- 172.21.15.78:0/1181257833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe644003a30 0x7fe644003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:26.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.233+0000 7fe65a7fc700 1 --2- 172.21.15.78:0/1181257833 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe65c1554e0 0x7fe644004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:26.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.233+0000 7fe65a7fc700 1 --2- 172.21.15.78:0/1181257833 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe644004d90 0x7fe644005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:26.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.233+0000 7fe65a7fc700 1 -- 172.21.15.78:0/1181257833 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65c155930 con 0x7fe644003a30 2023-10-14T11:58:26.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.233+0000 7fe65a7fc700 1 -- 172.21.15.78:0/1181257833 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64400bba0 con 0x7fe644004d90 2023-10-14T11:58:26.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.233+0000 7fe65a7fc700 1 -- 172.21.15.78:0/1181257833 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe64400bd30 con 0x7fe65c1554e0 2023-10-14T11:58:26.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.233+0000 7fe65b7fe700 1 --2- 172.21.15.78:0/1181257833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe644003a30 0x7fe644003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:26.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.233+0000 7fe65bfff700 1 --2- 172.21.15.78:0/1181257833 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe644004d90 0x7fe644005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:26.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.233+0000 7fe65affd700 1 --2- 172.21.15.78:0/1181257833 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe65c1554e0 0x7fe644004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:26.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.233+0000 7fe65b7fe700 1 -- 172.21.15.78:0/1181257833 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe65c1554e0 msgr2=0x7fe644004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:26.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.233+0000 7fe65b7fe700 1 --2- 172.21.15.78:0/1181257833 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe65c1554e0 0x7fe644004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:26.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.233+0000 7fe65b7fe700 1 -- 172.21.15.78:0/1181257833 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe644004d90 msgr2=0x7fe644005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:26.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.233+0000 7fe65b7fe700 1 --2- 172.21.15.78:0/1181257833 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe644004d90 0x7fe644005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:26.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.233+0000 7fe65b7fe700 1 -- 172.21.15.78:0/1181257833 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6440035b0 con 0x7fe644003a30 2023-10-14T11:58:26.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.233+0000 7fe65bfff700 1 --2- 172.21.15.78:0/1181257833 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe644004d90 0x7fe644005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:26.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.234+0000 7fe65b7fe700 1 --2- 172.21.15.78:0/1181257833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe644003a30 0x7fe644003e10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe64c00c320 tx=0x7fe644005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96a9b13a62c0907f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:26.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.234+0000 7fe65a7fc700 1 -- 172.21.15.78:0/1181257833 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64c00ce20 con 0x7fe644003a30 2023-10-14T11:58:26.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.234+0000 7fe65a7fc700 1 -- 172.21.15.78:0/1181257833 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe64c005960 con 0x7fe644003a30 2023-10-14T11:58:26.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.234+0000 7fe65a7fc700 1 -- 172.21.15.78:0/1181257833 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64c005d10 con 0x7fe644003a30 2023-10-14T11:58:26.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.234+0000 7fe661eed700 1 -- 172.21.15.78:0/1181257833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe644003a30 msgr2=0x7fe644003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:26.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.234+0000 7fe661eed700 1 --2- 172.21.15.78:0/1181257833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe644003a30 0x7fe644003e10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe64c00c320 tx=0x7fe644005880 comp rx=0 tx=0).stop 2023-10-14T11:58:26.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.234+0000 7fe65bfff700 1 -- 172.21.15.78:0/1181257833 reap_dead start 2023-10-14T11:58:26.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.234+0000 7fe661eed700 1 -- 172.21.15.78:0/1181257833 shutdown_connections 2023-10-14T11:58:26.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.234+0000 7fe661eed700 1 -- 172.21.15.78:0/1181257833 >> 172.21.15.78:0/1181257833 conn(0x7fe65c0c3ac0 msgr2=0x7fe65c0c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:26.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.234+0000 7fe661eed700 1 -- 172.21.15.78:0/1181257833 shutdown_connections 2023-10-14T11:58:26.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.234+0000 7fe661eed700 1 -- 172.21.15.78:0/1181257833 wait complete. 2023-10-14T11:58:26.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.236+0000 7fe661eed700 1 Processor -- start 2023-10-14T11:58:26.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.236+0000 7fe661eed700 1 -- start start 2023-10-14T11:58:26.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.237+0000 7fe661eed700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe644003a30 0x7fe65c0c2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:26.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.237+0000 7fe661eed700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe65c134400 0x7fe65c133af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:26.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.237+0000 7fe661eed700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe65c131710 0x7fe65c131bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:26.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.237+0000 7fe661eed700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65c1e9690 con 0x7fe644003a30 2023-10-14T11:58:26.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.237+0000 7fe661eed700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65c0c2780 con 0x7fe65c134400 2023-10-14T11:58:26.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.237+0000 7fe661eed700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe65c0c2e70 con 0x7fe65c131710 2023-10-14T11:58:26.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.237+0000 7fe65b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe644003a30 0x7fe65c0c2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:26.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.237+0000 7fe65b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe644003a30 0x7fe65c0c2240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45036/0 (socket says 172.21.15.78:45036) 2023-10-14T11:58:26.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.237+0000 7fe65b7fe700 1 -- 172.21.15.78:0/4269543472 learned_addr learned my addr 172.21.15.78:0/4269543472 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:26.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.237+0000 7fe65affd700 1 --2- 172.21.15.78:0/4269543472 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe65c134400 0x7fe65c133af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:26.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.237+0000 7fe65bfff700 1 --2- 172.21.15.78:0/4269543472 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe65c131710 0x7fe65c131bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:26.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.237+0000 7fe65b7fe700 1 -- 172.21.15.78:0/4269543472 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe65c131710 msgr2=0x7fe65c131bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:26.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.237+0000 7fe65b7fe700 1 --2- 172.21.15.78:0/4269543472 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe65c131710 0x7fe65c131bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:26.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.237+0000 7fe65b7fe700 1 -- 172.21.15.78:0/4269543472 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe65c134400 msgr2=0x7fe65c133af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:26.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.237+0000 7fe65b7fe700 1 --2- 172.21.15.78:0/4269543472 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe65c134400 0x7fe65c133af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:26.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.237+0000 7fe65b7fe700 1 -- 172.21.15.78:0/4269543472 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe65c0c3150 con 0x7fe644003a30 2023-10-14T11:58:26.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.237+0000 7fe65affd700 1 --2- 172.21.15.78:0/4269543472 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe65c134400 0x7fe65c133af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:26.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.237+0000 7fe65b7fe700 1 --2- 172.21.15.78:0/4269543472 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe644003a30 0x7fe65c0c2240 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe6440057f0 tx=0x7fe64c001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:26.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.237+0000 7fe643fff700 1 -- 172.21.15.78:0/4269543472 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64c007bc0 con 0x7fe644003a30 2023-10-14T11:58:26.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.237+0000 7fe643fff700 1 -- 172.21.15.78:0/4269543472 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe64c004870 con 0x7fe644003a30 2023-10-14T11:58:26.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.237+0000 7fe661eed700 1 -- 172.21.15.78:0/4269543472 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe65c0c3430 con 0x7fe644003a30 2023-10-14T11:58:26.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.238+0000 7fe643fff700 1 -- 172.21.15.78:0/4269543472 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64c004c20 con 0x7fe644003a30 2023-10-14T11:58:26.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.238+0000 7fe661eed700 1 -- 172.21.15.78:0/4269543472 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe65c0c2970 con 0x7fe644003a30 2023-10-14T11:58:26.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.238+0000 7fe661eed700 1 -- 172.21.15.78:0/4269543472 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe65c04ebb0 con 0x7fe644003a30 2023-10-14T11:58:26.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.239+0000 7fe643fff700 1 -- 172.21.15.78:0/4269543472 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe64c0070d0 con 0x7fe644003a30 2023-10-14T11:58:26.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.239+0000 7fe643fff700 1 --2- 172.21.15.78:0/4269543472 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe64806af30 0x7fe64806d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:26.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.239+0000 7fe643fff700 1 -- 172.21.15.78:0/4269543472 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe64c08abd0 con 0x7fe644003a30 2023-10-14T11:58:26.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.239+0000 7fe65affd700 1 --2- 172.21.15.78:0/4269543472 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe64806af30 0x7fe64806d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:26.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.240+0000 7fe65affd700 1 --2- 172.21.15.78:0/4269543472 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe64806af30 0x7fe64806d3f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe65c138080 tx=0x7fe650007d00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:26.255 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.244+0000 7fe643fff700 1 -- 172.21.15.78:0/4269543472 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe64c055d10 con 0x7fe644003a30 2023-10-14T11:58:26.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.477+0000 7fe661eed700 1 -- 172.21.15.78:0/4269543472 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe65c074b80 con 0x7fe644003a30 2023-10-14T11:58:26.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.477+0000 7fe643fff700 1 -- 172.21.15.78:0/4269543472 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4904 (secure 0 0 0) 0x7fe64c022090 con 0x7fe644003a30 2023-10-14T11:58:26.480 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:26.480 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/353043244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":353043244},{"type":"v1","addr":"172.21.15.78:6836","nonce":353043244}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6834/3389817343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":3389817343},{"type":"v1","addr":"172.21.15.195:6834","nonce":3389817343}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4267,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/460176701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6833","nonce":460176701},{"type":"v1","addr":"172.21.15.195:6835","nonce":460176701}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T11:58:14.535101+0000","modified":"2023-10-14T11:58:17.559202+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/920300787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":920300787},{"type":"v1","addr":"172.21.15.78:6837","nonce":920300787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-14T11:58:26.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.479+0000 7fe661eed700 1 -- 172.21.15.78:0/4269543472 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe64806af30 msgr2=0x7fe64806d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:26.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.479+0000 7fe661eed700 1 --2- 172.21.15.78:0/4269543472 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe64806af30 0x7fe64806d3f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe65c138080 tx=0x7fe650007d00 comp rx=0 tx=0).stop 2023-10-14T11:58:26.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.479+0000 7fe661eed700 1 -- 172.21.15.78:0/4269543472 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe644003a30 msgr2=0x7fe65c0c2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:26.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.479+0000 7fe661eed700 1 --2- 172.21.15.78:0/4269543472 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe644003a30 0x7fe65c0c2240 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe6440057f0 tx=0x7fe64c001a00 comp rx=0 tx=0).stop 2023-10-14T11:58:26.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.479+0000 7fe661eed700 1 -- 172.21.15.78:0/4269543472 shutdown_connections 2023-10-14T11:58:26.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.479+0000 7fe661eed700 1 --2- 172.21.15.78:0/4269543472 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe65c131710 0x7fe65c131bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:26.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.479+0000 7fe661eed700 1 --2- 172.21.15.78:0/4269543472 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe644003a30 0x7fe65c0c2240 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:26.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.479+0000 7fe661eed700 1 --2- 172.21.15.78:0/4269543472 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe64806af30 0x7fe64806d3f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:26.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.479+0000 7fe661eed700 1 --2- 172.21.15.78:0/4269543472 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe65c134400 0x7fe65c133af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:26.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.479+0000 7fe661eed700 1 -- 172.21.15.78:0/4269543472 >> 172.21.15.78:0/4269543472 conn(0x7fe65c0c3ac0 msgr2=0x7fe65c0667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:26.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.480+0000 7fe661eed700 1 -- 172.21.15.78:0/4269543472 shutdown_connections 2023-10-14T11:58:26.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.480+0000 7fe661eed700 1 -- 172.21.15.78:0/4269543472 wait complete. 2023-10-14T11:58:26.484 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 8 2023-10-14T11:58:26.494 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T11:58:26.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.611+0000 7fd380587700 1 Processor -- start 2023-10-14T11:58:26.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.612+0000 7fd380587700 1 -- start start 2023-10-14T11:58:26.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.612+0000 7fd380587700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd37806ab40 0x7fd37806f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:26.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.612+0000 7fd380587700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd3780c0350 0x7fd37813d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:26.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.612+0000 7fd380587700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd37806b630 con 0x7fd3781554e0 2023-10-14T11:58:26.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.612+0000 7fd380587700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3780c49f0 con 0x7fd37806ab40 2023-10-14T11:58:26.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.612+0000 7fd380587700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3780726a0 con 0x7fd3780c0350 2023-10-14T11:58:26.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.613+0000 7fd37e323700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd37806ab40 0x7fd37806f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:26.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.613+0000 7fd37e323700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd37806ab40 0x7fd37806f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45044/0 (socket says 172.21.15.78:45044) 2023-10-14T11:58:26.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.613+0000 7fd37e323700 1 -- 172.21.15.78:0/1250033694 learned_addr learned my addr 172.21.15.78:0/1250033694 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:26.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.613+0000 7fd37eb24700 1 --2- 172.21.15.78:0/1250033694 >> v2:172.21.15.195:3300/0 conn(0x7fd3780c0350 0x7fd37813d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:26.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.613+0000 7fd37e323700 1 -- 172.21.15.78:0/1250033694 >> v2:172.21.15.195:3300/0 conn(0x7fd3780c0350 msgr2=0x7fd37813d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:26.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.613+0000 7fd37e323700 1 --2- 172.21.15.78:0/1250033694 >> v2:172.21.15.195:3300/0 conn(0x7fd3780c0350 0x7fd37813d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:26.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.613+0000 7fd37e323700 1 -- 172.21.15.78:0/1250033694 >> v1:172.21.15.195:6789/0 conn(0x7fd3781554e0 legacy=0x7fd37806af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:26.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.613+0000 7fd37e323700 1 -- 172.21.15.78:0/1250033694 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd378072980 con 0x7fd37806ab40 2023-10-14T11:58:26.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.613+0000 7fd37e323700 1 --2- 172.21.15.78:0/1250033694 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd37806ab40 0x7fd37806f1e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd3780c1860 tx=0x7fd368009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=291a117b387bac03 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:26.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.613+0000 7fd37d321700 1 -- 172.21.15.78:0/1250033694 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd368004230 con 0x7fd37806ab40 2023-10-14T11:58:26.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.613+0000 7fd37d321700 1 -- 172.21.15.78:0/1250033694 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd368004390 con 0x7fd37806ab40 2023-10-14T11:58:26.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.613+0000 7fd37d321700 1 -- 172.21.15.78:0/1250033694 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3680036a0 con 0x7fd37806ab40 2023-10-14T11:58:26.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.614+0000 7fd380587700 1 -- 172.21.15.78:0/1250033694 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd37806ab40 msgr2=0x7fd37806f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:26.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.614+0000 7fd380587700 1 --2- 172.21.15.78:0/1250033694 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd37806ab40 0x7fd37806f1e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd3780c1860 tx=0x7fd368009bd0 comp rx=0 tx=0).stop 2023-10-14T11:58:26.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.614+0000 7fd380587700 1 -- 172.21.15.78:0/1250033694 shutdown_connections 2023-10-14T11:58:26.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.614+0000 7fd380587700 1 --2- 172.21.15.78:0/1250033694 >> v2:172.21.15.195:3300/0 conn(0x7fd3780c0350 0x7fd37813d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:26.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.614+0000 7fd380587700 1 --2- 172.21.15.78:0/1250033694 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd37806ab40 0x7fd37806f1e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:26.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.614+0000 7fd380587700 1 -- 172.21.15.78:0/1250033694 >> 172.21.15.78:0/1250033694 conn(0x7fd3780c3ac0 msgr2=0x7fd3780c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:26.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.614+0000 7fd380587700 1 -- 172.21.15.78:0/1250033694 shutdown_connections 2023-10-14T11:58:26.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.614+0000 7fd380587700 1 -- 172.21.15.78:0/1250033694 wait complete. 2023-10-14T11:58:26.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.615+0000 7fd380587700 1 Processor -- start 2023-10-14T11:58:26.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.615+0000 7fd380587700 1 -- start start 2023-10-14T11:58:26.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.615+0000 7fd380587700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd37806ab40 0x7fd3780ab4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:26.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.616+0000 7fd380587700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3780c0350 0x7fd3781ecb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:26.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.616+0000 7fd380587700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3781554e0 0x7fd3781ed0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:26.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.616+0000 7fd380587700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3780c1120 con 0x7fd3781554e0 2023-10-14T11:58:26.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.616+0000 7fd380587700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3780ab9f0 con 0x7fd3780c0350 2023-10-14T11:58:26.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.616+0000 7fd380587700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3781ed6d0 con 0x7fd37806ab40 2023-10-14T11:58:26.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.616+0000 7fd37eb24700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3781554e0 0x7fd3781ed0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:26.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.616+0000 7fd37eb24700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3781554e0 0x7fd3781ed0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45054/0 (socket says 172.21.15.78:45054) 2023-10-14T11:58:26.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.616+0000 7fd37eb24700 1 -- 172.21.15.78:0/1809687922 learned_addr learned my addr 172.21.15.78:0/1809687922 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:26.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.616+0000 7fd37db22700 1 --2- 172.21.15.78:0/1809687922 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3780c0350 0x7fd3781ecb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:26.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.616+0000 7fd37e323700 1 --2- 172.21.15.78:0/1809687922 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd37806ab40 0x7fd3780ab4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:26.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.616+0000 7fd37eb24700 1 -- 172.21.15.78:0/1809687922 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd37806ab40 msgr2=0x7fd3780ab4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:26.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.616+0000 7fd37eb24700 1 --2- 172.21.15.78:0/1809687922 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd37806ab40 0x7fd3780ab4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:26.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.616+0000 7fd37eb24700 1 -- 172.21.15.78:0/1809687922 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3780c0350 msgr2=0x7fd3781ecb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:26.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.616+0000 7fd37eb24700 1 --2- 172.21.15.78:0/1809687922 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3780c0350 0x7fd3781ecb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:26.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.616+0000 7fd37eb24700 1 -- 172.21.15.78:0/1809687922 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd368009550 con 0x7fd3781554e0 2023-10-14T11:58:26.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.616+0000 7fd37eb24700 1 --2- 172.21.15.78:0/1809687922 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3781554e0 0x7fd3781ed0d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd37000cd70 tx=0x7fd37000a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:26.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.616+0000 7fd36effd700 1 -- 172.21.15.78:0/1809687922 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd370003a20 con 0x7fd3781554e0 2023-10-14T11:58:26.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.616+0000 7fd36effd700 1 -- 172.21.15.78:0/1809687922 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd370003b80 con 0x7fd3781554e0 2023-10-14T11:58:26.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.616+0000 7fd380587700 1 -- 172.21.15.78:0/1809687922 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3780abf90 con 0x7fd3781554e0 2023-10-14T11:58:26.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.616+0000 7fd36effd700 1 -- 172.21.15.78:0/1809687922 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd370004020 con 0x7fd3781554e0 2023-10-14T11:58:26.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.616+0000 7fd380587700 1 -- 172.21.15.78:0/1809687922 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3780c2c90 con 0x7fd3781554e0 2023-10-14T11:58:26.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.617+0000 7fd380587700 1 -- 172.21.15.78:0/1809687922 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd378074b80 con 0x7fd3781554e0 2023-10-14T11:58:26.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.618+0000 7fd36effd700 1 -- 172.21.15.78:0/1809687922 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd370004180 con 0x7fd3781554e0 2023-10-14T11:58:26.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.618+0000 7fd36effd700 1 --2- 172.21.15.78:0/1809687922 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd35c06af00 0x7fd35c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:26.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.618+0000 7fd37e323700 1 --2- 172.21.15.78:0/1809687922 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd35c06af00 0x7fd35c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:26.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.618+0000 7fd36effd700 1 -- 172.21.15.78:0/1809687922 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd370012070 con 0x7fd3781554e0 2023-10-14T11:58:26.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.618+0000 7fd37e323700 1 --2- 172.21.15.78:0/1809687922 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd35c06af00 0x7fd35c06d3c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd368009130 tx=0x7fd368008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:26.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.622+0000 7fd36effd700 1 -- 172.21.15.78:0/1809687922 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd370054d00 con 0x7fd3781554e0 2023-10-14T11:58:26.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.811+0000 7fd380587700 1 -- 172.21.15.78:0/1809687922 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd378154cb0 con 0x7fd3781554e0 2023-10-14T11:58:26.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.813+0000 7fd36effd700 1 -- 172.21.15.78:0/1809687922 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16481 (secure 0 0 0) 0x7fd370058320 con 0x7fd3781554e0 2023-10-14T11:58:26.815 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:26.815 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":21,"fsid":"4c5dc134-3092-406a-a1e1-13e6d9e926e8","created":"2023-10-14T11:57:57.363750+0000","modified":"2023-10-14T11:58:16.139413+0000","last_up_change":"2023-10-14T11:58:06.982472+0000","last_in_change":"2023-10-14T11:57:59.821373+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T11:58:09.001200+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T11:58:12.778573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T11:58:13.490286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"625a5fc0-9474-4eae-b80e-14c27a74ad19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6801","nonce":3884969780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6803","nonce":3884969780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6807","nonce":3884969780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6805","nonce":3884969780}]},"public_addr":"172.21.15.78:6801/3884969780","cluster_addr":"172.21.15.78:6803/3884969780","heartbeat_back_addr":"172.21.15.78:6807/3884969780","heartbeat_front_addr":"172.21.15.78:6805/3884969780","state":["exists","up"]},{"osd":1,"uuid":"49d63db1-2981-49ab-9c40-c50fc744b982","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6809","nonce":2176928406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6811","nonce":2176928406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6815","nonce":2176928406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6813","nonce":2176928406}]},"public_addr":"172.21.15.78:6809/2176928406","cluster_addr":"172.21.15.78:6811/2176928406","heartbeat_back_addr":"172.21.15.78:6815/2176928406","heartbeat_front_addr":"172.21.15.78:6813/2176928406","state":["exists","up"]},{"osd":2,"uuid":"1d7d0b29-cbd1-4717-9fe6-765a8d4998b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6825","nonce":3647044577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6827","nonce":3647044577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6831","nonce":3647044577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6829","nonce":3647044577}]},"public_addr":"172.21.15.78:6825/3647044577","cluster_addr":"172.21.15.78:6827/3647044577","heartbeat_back_addr":"172.21.15.78:6831/3647044577","heartbeat_front_addr":"172.21.15.78:6829/3647044577","state":["exists","up"]},{"osd":3,"uuid":"2f75ded1-e9e2-40e7-99ae-45e1e425e1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6817","nonce":3584112288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6819","nonce":3584112288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6823","nonce":3584112288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6821","nonce":3584112288}]},"public_addr":"172.21.15.78:6817/3584112288","cluster_addr":"172.21.15.78:6819/3584112288","heartbeat_back_addr":"172.21.15.78:6823/3584112288","heartbeat_front_addr":"172.21.15.78:6821/3584112288","state":["exists","up"]},{"osd":4,"uuid":"5130b432-8358-46cb-8a31-3064c976f6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6801","nonce":3732756301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6803","nonce":3732756301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6807","nonce":3732756301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6805","nonce":3732756301}]},"public_addr":"172.21.15.195:6801/3732756301","cluster_addr":"172.21.15.195:6803/3732756301","heartbeat_back_addr":"172.21.15.195:6807/3732756301","heartbeat_front_addr":"172.21.15.195:6805/3732756301","state":["exists","up"]},{"osd":5,"uuid":"60e234a9-a506-4e40-9c19-49a58f3cf28c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6825","nonce":2857706411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6827","nonce":2857706411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6831","nonce":2857706411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6829","nonce":2857706411}]},"public_addr":"172.21.15.195:6825/2857706411","cluster_addr":"172.21.15.195:6827/2857706411","heartbeat_back_addr":"172.21.15.195:6831/2857706411","heartbeat_front_addr":"172.21.15.195:6829/2857706411","state":["exists","up"]},{"osd":6,"uuid":"df7d2dd6-26b6-4eb1-96e5-94d1f3955b33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6810","nonce":72554374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6813","nonce":72554374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6820","nonce":72554374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6817","nonce":72554374}]},"public_addr":"172.21.15.195:6810/72554374","cluster_addr":"172.21.15.195:6813/72554374","heartbeat_back_addr":"172.21.15.195:6820/72554374","heartbeat_front_addr":"172.21.15.195:6817/72554374","state":["exists","up"]},{"osd":7,"uuid":"47a9d4de-f56d-4aa4-9900-c4a9cf9b2f97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6809","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6811","nonce":482472979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6816","nonce":482472979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6823","nonce":482472979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6821","nonce":482472979}]},"public_addr":"172.21.15.195:6811/482472979","cluster_addr":"172.21.15.195:6816/482472979","heartbeat_back_addr":"172.21.15.195:6823/482472979","heartbeat_front_addr":"172.21.15.195:6821/482472979","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.754003+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.005994+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.043603+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.024506+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.062585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.033767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.994825+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.038300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T11:58:26.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.815+0000 7fd380587700 1 -- 172.21.15.78:0/1809687922 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd35c06af00 msgr2=0x7fd35c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:26.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.815+0000 7fd380587700 1 --2- 172.21.15.78:0/1809687922 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd35c06af00 0x7fd35c06d3c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd368009130 tx=0x7fd368008030 comp rx=0 tx=0).stop 2023-10-14T11:58:26.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.815+0000 7fd380587700 1 -- 172.21.15.78:0/1809687922 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3781554e0 msgr2=0x7fd3781ed0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:26.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.815+0000 7fd380587700 1 --2- 172.21.15.78:0/1809687922 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3781554e0 0x7fd3781ed0d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd37000cd70 tx=0x7fd37000a570 comp rx=0 tx=0).stop 2023-10-14T11:58:26.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.815+0000 7fd380587700 1 -- 172.21.15.78:0/1809687922 shutdown_connections 2023-10-14T11:58:26.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.815+0000 7fd380587700 1 --2- 172.21.15.78:0/1809687922 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3781554e0 0x7fd3781ed0d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:26.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.815+0000 7fd380587700 1 --2- 172.21.15.78:0/1809687922 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd37806ab40 0x7fd3780ab4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:26.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.815+0000 7fd380587700 1 --2- 172.21.15.78:0/1809687922 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd35c06af00 0x7fd35c06d3c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:26.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.815+0000 7fd380587700 1 --2- 172.21.15.78:0/1809687922 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3780c0350 0x7fd3781ecb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:26.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.815+0000 7fd380587700 1 -- 172.21.15.78:0/1809687922 >> 172.21.15.78:0/1809687922 conn(0x7fd3780c3ac0 msgr2=0x7fd37813cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:26.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.815+0000 7fd380587700 1 -- 172.21.15.78:0/1809687922 shutdown_connections 2023-10-14T11:58:26.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.815+0000 7fd380587700 1 -- 172.21.15.78:0/1809687922 wait complete. 2023-10-14T11:58:26.829 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T11:58:09.001200+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T11:58:12.778573+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T11:58:13.490286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T11:58:26.829 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T11:58:26.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.950+0000 7f80741fb700 1 Processor -- start 2023-10-14T11:58:26.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.950+0000 7f80741fb700 1 -- start start 2023-10-14T11:58:26.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.951+0000 7f80741fb700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f806c0c4540 0x7f806c0c02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:26.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.951+0000 7f80741fb700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f806c06b500 0x7f806c0c0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:26.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.951+0000 7f80741fb700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f806c06ebe0 con 0x7f806c13dbe0 2023-10-14T11:58:26.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.951+0000 7f80741fb700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f806c06d330 con 0x7f806c06b500 2023-10-14T11:58:26.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.951+0000 7f80741fb700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f806c0c0d60 con 0x7f806c0c4540 2023-10-14T11:58:26.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.951+0000 7f8071f97700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f806c13dbe0 0x7f806c06a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49934/0 (socket says 172.21.15.78:49934) 2023-10-14T11:58:26.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.951+0000 7f8071f97700 1 -- 172.21.15.78:0/334831898 learned_addr learned my addr 172.21.15.78:0/334831898 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:26.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.951+0000 7f8072798700 1 --2- 172.21.15.78:0/334831898 >> v2:172.21.15.195:3300/0 conn(0x7f806c06b500 0x7f806c0c0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:26.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.951+0000 7f8071796700 1 --2- 172.21.15.78:0/334831898 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f806c0c4540 0x7f806c0c02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:26.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.951+0000 7f8070f95700 1 -- 172.21.15.78:0/334831898 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2911536367 0 0) 0x7f806c06ebe0 con 0x7f806c13dbe0 2023-10-14T11:58:26.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.951+0000 7f8070f95700 1 -- 172.21.15.78:0/334831898 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80580035b0 con 0x7f806c13dbe0 2023-10-14T11:58:26.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.951+0000 7f8070f95700 1 -- 172.21.15.78:0/334831898 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3638577194 0 0) 0x7f80580035b0 con 0x7f806c13dbe0 2023-10-14T11:58:26.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.951+0000 7f8070f95700 1 -- 172.21.15.78:0/334831898 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f806c0c4540 msgr2=0x7f806c0c02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:26.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.951+0000 7f8070f95700 1 --2- 172.21.15.78:0/334831898 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f806c0c4540 0x7f806c0c02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:26.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.951+0000 7f8070f95700 1 -- 172.21.15.78:0/334831898 >> v2:172.21.15.195:3300/0 conn(0x7f806c06b500 msgr2=0x7f806c0c0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:26.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.951+0000 7f8070f95700 1 --2- 172.21.15.78:0/334831898 >> v2:172.21.15.195:3300/0 conn(0x7f806c06b500 0x7f806c0c0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:26.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.951+0000 7f8070f95700 1 -- 172.21.15.78:0/334831898 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f806c1558c0 con 0x7f806c13dbe0 2023-10-14T11:58:26.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.951+0000 7f8070f95700 1 -- 172.21.15.78:0/334831898 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441799917 0 0) 0x7f8068001670 con 0x7f806c13dbe0 2023-10-14T11:58:26.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.951+0000 7f8070f95700 1 -- 172.21.15.78:0/334831898 >> v1:172.21.15.78:6789/0 conn(0x7f806c13dbe0 legacy=0x7f806c06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:26.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.951+0000 7f8070f95700 1 --2- 172.21.15.78:0/334831898 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8058003a00 0x7f8058003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:26.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.951+0000 7f8070f95700 1 --2- 172.21.15.78:0/334831898 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8058004850 0x7f8058004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:26.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.951+0000 7f8070f95700 1 --2- 172.21.15.78:0/334831898 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f806c0c4540 0x7f8058005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:26.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.951+0000 7f8070f95700 1 -- 172.21.15.78:0/334831898 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806c06d330 con 0x7f8058003a00 2023-10-14T11:58:26.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.951+0000 7f8070f95700 1 -- 172.21.15.78:0/334831898 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806c0c0d60 con 0x7f8058004850 2023-10-14T11:58:26.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.951+0000 7f8070f95700 1 -- 172.21.15.78:0/334831898 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f805800ba10 con 0x7f806c0c4540 2023-10-14T11:58:26.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.952+0000 7f8071f97700 1 --2- 172.21.15.78:0/334831898 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8058003a00 0x7f8058003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:26.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.952+0000 7f8071796700 1 --2- 172.21.15.78:0/334831898 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8058004850 0x7f8058004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:26.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.952+0000 7f8072798700 1 --2- 172.21.15.78:0/334831898 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f806c0c4540 0x7f8058005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:26.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.952+0000 7f8071f97700 1 -- 172.21.15.78:0/334831898 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f806c0c4540 msgr2=0x7f8058005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:26.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.952+0000 7f8071f97700 1 --2- 172.21.15.78:0/334831898 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f806c0c4540 0x7f8058005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:26.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.952+0000 7f8071f97700 1 -- 172.21.15.78:0/334831898 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8058004850 msgr2=0x7f8058004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:26.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.952+0000 7f8071f97700 1 --2- 172.21.15.78:0/334831898 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8058004850 0x7f8058004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:26.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.952+0000 7f8071f97700 1 -- 172.21.15.78:0/334831898 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80580035b0 con 0x7f8058003a00 2023-10-14T11:58:26.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.952+0000 7f8072798700 1 --2- 172.21.15.78:0/334831898 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f806c0c4540 0x7f8058005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:26.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.952+0000 7f8071f97700 1 --2- 172.21.15.78:0/334831898 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8058003a00 0x7f8058003e20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f806800c320 tx=0x7f8058005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b842e0507c20d2f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:26.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.952+0000 7f8070f95700 1 -- 172.21.15.78:0/334831898 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f806800ce60 con 0x7f8058003a00 2023-10-14T11:58:26.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.952+0000 7f8070f95700 1 -- 172.21.15.78:0/334831898 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8068005960 con 0x7f8058003a00 2023-10-14T11:58:26.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.952+0000 7f8070f95700 1 -- 172.21.15.78:0/334831898 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8068005cf0 con 0x7f8058003a00 2023-10-14T11:58:26.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.953+0000 7f80741fb700 1 -- 172.21.15.78:0/334831898 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8058003a00 msgr2=0x7f8058003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:26.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.953+0000 7f80741fb700 1 --2- 172.21.15.78:0/334831898 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8058003a00 0x7f8058003e20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f806800c320 tx=0x7f8058005880 comp rx=0 tx=0).stop 2023-10-14T11:58:26.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.953+0000 7f8072798700 1 -- 172.21.15.78:0/334831898 reap_dead start 2023-10-14T11:58:26.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.953+0000 7f80741fb700 1 -- 172.21.15.78:0/334831898 shutdown_connections 2023-10-14T11:58:26.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.953+0000 7f80741fb700 1 -- 172.21.15.78:0/334831898 >> 172.21.15.78:0/334831898 conn(0x7f806c0c3a50 msgr2=0x7f806c0c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:26.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.953+0000 7f80741fb700 1 -- 172.21.15.78:0/334831898 shutdown_connections 2023-10-14T11:58:26.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.953+0000 7f80741fb700 1 -- 172.21.15.78:0/334831898 wait complete. 2023-10-14T11:58:26.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.954+0000 7f80741fb700 1 Processor -- start 2023-10-14T11:58:26.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.954+0000 7f80741fb700 1 -- start start 2023-10-14T11:58:26.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.954+0000 7f80741fb700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8058003a00 0x7f806c1323f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:26.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.954+0000 7f80741fb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f806c133ad0 0x7f806c153e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:26.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.954+0000 7f80741fb700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f806c1edad0 0x7f806c1e98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:26.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.954+0000 7f80741fb700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806c132cb0 con 0x7f806c133ad0 2023-10-14T11:58:26.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.954+0000 7f80741fb700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806c1ed640 con 0x7f806c1edad0 2023-10-14T11:58:26.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.954+0000 7f80741fb700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f806c1329d0 con 0x7f8058003a00 2023-10-14T11:58:26.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.955+0000 7f8071796700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f806c133ad0 0x7f806c153e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:26.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.955+0000 7f8071796700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f806c133ad0 0x7f806c153e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45072/0 (socket says 172.21.15.78:45072) 2023-10-14T11:58:26.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.955+0000 7f8071796700 1 -- 172.21.15.78:0/2939940772 learned_addr learned my addr 172.21.15.78:0/2939940772 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:26.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.955+0000 7f8071f97700 1 --2- 172.21.15.78:0/2939940772 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8058003a00 0x7f806c1323f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:26.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.955+0000 7f8072798700 1 --2- 172.21.15.78:0/2939940772 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f806c1edad0 0x7f806c1e98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:26.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.955+0000 7f8071796700 1 -- 172.21.15.78:0/2939940772 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8058003a00 msgr2=0x7f806c1323f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:26.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.955+0000 7f8071796700 1 --2- 172.21.15.78:0/2939940772 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8058003a00 0x7f806c1323f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:26.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.955+0000 7f8071796700 1 -- 172.21.15.78:0/2939940772 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f806c1edad0 msgr2=0x7f806c1e98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:26.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.955+0000 7f8071796700 1 --2- 172.21.15.78:0/2939940772 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f806c1edad0 0x7f806c1e98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:26.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.955+0000 7f8071796700 1 -- 172.21.15.78:0/2939940772 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f806c1ecff0 con 0x7f806c133ad0 2023-10-14T11:58:26.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.955+0000 7f8071796700 1 --2- 172.21.15.78:0/2939940772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f806c133ad0 0x7f806c153e50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f805c011940 tx=0x7f805c011c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:26.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.955+0000 7f80627fc700 1 -- 172.21.15.78:0/2939940772 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805c004240 con 0x7f806c133ad0 2023-10-14T11:58:26.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.955+0000 7f80627fc700 1 -- 172.21.15.78:0/2939940772 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f805c0043a0 con 0x7f806c133ad0 2023-10-14T11:58:26.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.955+0000 7f80741fb700 1 -- 172.21.15.78:0/2939940772 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f806c1ed2d0 con 0x7f806c133ad0 2023-10-14T11:58:26.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.955+0000 7f80627fc700 1 -- 172.21.15.78:0/2939940772 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805c0038c0 con 0x7f806c133ad0 2023-10-14T11:58:26.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.955+0000 7f80741fb700 1 -- 172.21.15.78:0/2939940772 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f806c134be0 con 0x7f806c133ad0 2023-10-14T11:58:26.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.956+0000 7f80741fb700 1 -- 172.21.15.78:0/2939940772 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f806c134ec0 con 0x7f806c133ad0 2023-10-14T11:58:26.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.957+0000 7f80627fc700 1 -- 172.21.15.78:0/2939940772 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f805c003a20 con 0x7f806c133ad0 2023-10-14T11:58:26.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.957+0000 7f80627fc700 1 --2- 172.21.15.78:0/2939940772 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f805006af00 0x7f805006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:26.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.957+0000 7f80627fc700 1 -- 172.21.15.78:0/2939940772 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f805c089950 con 0x7f806c133ad0 2023-10-14T11:58:26.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.957+0000 7f8071f97700 1 --2- 172.21.15.78:0/2939940772 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f805006af00 0x7f805006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:26.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.957+0000 7f8071f97700 1 --2- 172.21.15.78:0/2939940772 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f805006af00 0x7f805006d3c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8068008fd0 tx=0x7f806800a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:26.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:26.961+0000 7f80627fc700 1 -- 172.21.15.78:0/2939940772 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f805c054a90 con 0x7f806c133ad0 2023-10-14T11:58:27.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.151+0000 7f80741fb700 1 -- 172.21.15.78:0/2939940772 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f806c1ece50 con 0x7f806c133ad0 2023-10-14T11:58:27.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.151+0000 7f80627fc700 1 -- 172.21.15.78:0/2939940772 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f806c1ece50 con 0x7f806c133ad0 2023-10-14T11:58:27.153 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-10-14T11:58:27.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.153+0000 7f80741fb700 1 -- 172.21.15.78:0/2939940772 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f805006af00 msgr2=0x7f805006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:27.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.153+0000 7f80741fb700 1 --2- 172.21.15.78:0/2939940772 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f805006af00 0x7f805006d3c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8068008fd0 tx=0x7f806800a000 comp rx=0 tx=0).stop 2023-10-14T11:58:27.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.153+0000 7f80741fb700 1 -- 172.21.15.78:0/2939940772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f806c133ad0 msgr2=0x7f806c153e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:27.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.153+0000 7f80741fb700 1 --2- 172.21.15.78:0/2939940772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f806c133ad0 0x7f806c153e50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f805c011940 tx=0x7f805c011c50 comp rx=0 tx=0).stop 2023-10-14T11:58:27.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.153+0000 7f80741fb700 1 -- 172.21.15.78:0/2939940772 shutdown_connections 2023-10-14T11:58:27.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.153+0000 7f80741fb700 1 --2- 172.21.15.78:0/2939940772 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f805006af00 0x7f805006d3c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:27.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.153+0000 7f80741fb700 1 --2- 172.21.15.78:0/2939940772 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f806c1edad0 0x7f806c1e98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:27.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.153+0000 7f80741fb700 1 --2- 172.21.15.78:0/2939940772 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8058003a00 0x7f806c1323f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:27.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.153+0000 7f80741fb700 1 --2- 172.21.15.78:0/2939940772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f806c133ad0 0x7f806c153e50 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:27.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.153+0000 7f80741fb700 1 -- 172.21.15.78:0/2939940772 >> 172.21.15.78:0/2939940772 conn(0x7f806c0c3a50 msgr2=0x7f806c13b620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:27.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.153+0000 7f80741fb700 1 -- 172.21.15.78:0/2939940772 shutdown_connections 2023-10-14T11:58:27.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.153+0000 7f80741fb700 1 -- 172.21.15.78:0/2939940772 wait complete. 2023-10-14T11:58:27.167 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T11:58:27.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.289+0000 7f0418a06700 1 Processor -- start 2023-10-14T11:58:27.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.289+0000 7f0418a06700 1 -- start start 2023-10-14T11:58:27.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.289+0000 7f0418a06700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f041413dbe0 0x7f041406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:27.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.290+0000 7f0418a06700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f041406ebe0 con 0x7f041406ae10 2023-10-14T11:58:27.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.290+0000 7f0418a06700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04141558c0 con 0x7f04140c4540 2023-10-14T11:58:27.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.290+0000 7f0418a06700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f041406d330 con 0x7f041413dbe0 2023-10-14T11:58:27.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.290+0000 7f0412d9d700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f041406ae10 0x7f041415f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49958/0 (socket says 172.21.15.78:49958) 2023-10-14T11:58:27.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.290+0000 7f0412d9d700 1 -- 172.21.15.78:0/1395729979 learned_addr learned my addr 172.21.15.78:0/1395729979 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:27.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.290+0000 7f041259c700 1 --2- 172.21.15.78:0/1395729979 >> v2:172.21.15.195:3300/0 conn(0x7f041413dbe0 0x7f041406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:27.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.290+0000 7f041159a700 1 -- 172.21.15.78:0/1395729979 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3386408261 0 0) 0x7f041406ebe0 con 0x7f041406ae10 2023-10-14T11:58:27.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.290+0000 7f041159a700 1 -- 172.21.15.78:0/1395729979 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03fc0035b0 con 0x7f041406ae10 2023-10-14T11:58:27.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.290+0000 7f041159a700 1 -- 172.21.15.78:0/1395729979 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1296856830 0 0) 0x7f04141558c0 con 0x7f04140c4540 2023-10-14T11:58:27.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.290+0000 7f041159a700 1 -- 172.21.15.78:0/1395729979 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f041406ebe0 con 0x7f04140c4540 2023-10-14T11:58:27.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.290+0000 7f041159a700 1 -- 172.21.15.78:0/1395729979 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3787402245 0 0) 0x7f03fc0035b0 con 0x7f041406ae10 2023-10-14T11:58:27.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.290+0000 7f041159a700 1 -- 172.21.15.78:0/1395729979 >> v2:172.21.15.195:3300/0 conn(0x7f041413dbe0 msgr2=0x7f041406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:27.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.290+0000 7f041159a700 1 --2- 172.21.15.78:0/1395729979 >> v2:172.21.15.195:3300/0 conn(0x7f041413dbe0 0x7f041406a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:27.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.290+0000 7f041159a700 1 -- 172.21.15.78:0/1395729979 >> v1:172.21.15.195:6789/0 conn(0x7f04140c4540 legacy=0x7f04140c02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:27.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.290+0000 7f041159a700 1 -- 172.21.15.78:0/1395729979 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0414154b00 con 0x7f041406ae10 2023-10-14T11:58:27.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.290+0000 7f041159a700 1 -- 172.21.15.78:0/1395729979 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441799917 0 0) 0x7f040c0019f0 con 0x7f041406ae10 2023-10-14T11:58:27.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.290+0000 7f041259c700 1 --2- 172.21.15.78:0/1395729979 >> v2:172.21.15.195:3300/0 conn(0x7f041413dbe0 0x7f041406a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T11:58:27.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.290+0000 7f041159a700 1 -- 172.21.15.78:0/1395729979 >> v1:172.21.15.78:6789/0 conn(0x7f041406ae10 legacy=0x7f041415f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:27.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.290+0000 7f041159a700 1 --2- 172.21.15.78:0/1395729979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03fc003b90 0x7f03fc003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:27.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.290+0000 7f041159a700 1 --2- 172.21.15.78:0/1395729979 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f03fc004850 0x7f03fc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:27.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.290+0000 7f041159a700 1 --2- 172.21.15.78:0/1395729979 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03fc005210 0x7f03fc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:27.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.290+0000 7f041159a700 1 -- 172.21.15.78:0/1395729979 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f041406d330 con 0x7f03fc003b90 2023-10-14T11:58:27.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.290+0000 7f041159a700 1 -- 172.21.15.78:0/1395729979 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03fc00c090 con 0x7f03fc005210 2023-10-14T11:58:27.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.290+0000 7f041159a700 1 -- 172.21.15.78:0/1395729979 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03fc00c250 con 0x7f03fc004850 2023-10-14T11:58:27.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.290+0000 7f041259c700 1 --2- 172.21.15.78:0/1395729979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03fc003b90 0x7f03fc003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:27.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.290+0000 7f0412d9d700 1 --2- 172.21.15.78:0/1395729979 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03fc005210 0x7f03fc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:27.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.291+0000 7f0411d9b700 1 --2- 172.21.15.78:0/1395729979 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f03fc004850 0x7f03fc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:27.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.291+0000 7f041259c700 1 -- 172.21.15.78:0/1395729979 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f03fc004850 msgr2=0x7f03fc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:27.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.291+0000 7f041259c700 1 --2- 172.21.15.78:0/1395729979 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f03fc004850 0x7f03fc004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:27.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.291+0000 7f041259c700 1 -- 172.21.15.78:0/1395729979 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03fc005210 msgr2=0x7f03fc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:27.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.291+0000 7f041259c700 1 --2- 172.21.15.78:0/1395729979 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03fc005210 0x7f03fc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:27.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.291+0000 7f041259c700 1 -- 172.21.15.78:0/1395729979 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03fc0035b0 con 0x7f03fc003b90 2023-10-14T11:58:27.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.291+0000 7f0412d9d700 1 --2- 172.21.15.78:0/1395729979 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03fc005210 0x7f03fc00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:27.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.291+0000 7f0412d9d700 1 -- 172.21.15.78:0/1395729979 reap_dead start 2023-10-14T11:58:27.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.291+0000 7f041259c700 1 --2- 172.21.15.78:0/1395729979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03fc003b90 0x7f03fc003f90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0404009d30 tx=0x7f03fc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3556121e76279253 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:27.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.291+0000 7f041159a700 1 -- 172.21.15.78:0/1395729979 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0404009e80 con 0x7f03fc003b90 2023-10-14T11:58:27.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.291+0000 7f041159a700 1 -- 172.21.15.78:0/1395729979 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f040400e9a0 con 0x7f03fc003b90 2023-10-14T11:58:27.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.291+0000 7f041159a700 1 -- 172.21.15.78:0/1395729979 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040400ed10 con 0x7f03fc003b90 2023-10-14T11:58:27.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.291+0000 7f0418a06700 1 -- 172.21.15.78:0/1395729979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03fc003b90 msgr2=0x7f03fc003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:27.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.291+0000 7f0418a06700 1 --2- 172.21.15.78:0/1395729979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03fc003b90 0x7f03fc003f90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0404009d30 tx=0x7f03fc005880 comp rx=0 tx=0).stop 2023-10-14T11:58:27.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.291+0000 7f0418a06700 1 -- 172.21.15.78:0/1395729979 shutdown_connections 2023-10-14T11:58:27.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.291+0000 7f0418a06700 1 --2- 172.21.15.78:0/1395729979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03fc003b90 0x7f03fc003f90 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:27.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.291+0000 7f0418a06700 1 -- 172.21.15.78:0/1395729979 >> 172.21.15.78:0/1395729979 conn(0x7f04140c3a50 msgr2=0x7f04140c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:27.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.291+0000 7f0418a06700 1 -- 172.21.15.78:0/1395729979 shutdown_connections 2023-10-14T11:58:27.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.291+0000 7f0418a06700 1 -- 172.21.15.78:0/1395729979 wait complete. 2023-10-14T11:58:27.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.293+0000 7f0418a06700 1 Processor -- start 2023-10-14T11:58:27.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.293+0000 7f0418a06700 1 -- start start 2023-10-14T11:58:27.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.293+0000 7f0418a06700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f03fc003b90 0x7f04141ea2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:27.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.293+0000 7f0418a06700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0414158860 0x7f04141507d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:27.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.293+0000 7f0418a06700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0414150d10 0x7f04141573b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:27.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.293+0000 7f0418a06700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04141ea830 con 0x7f0414150d10 2023-10-14T11:58:27.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.293+0000 7f0418a06700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0414135fb0 con 0x7f0414158860 2023-10-14T11:58:27.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.293+0000 7f0418a06700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0414156260 con 0x7f03fc003b90 2023-10-14T11:58:27.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.293+0000 7f0412d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0414150d10 0x7f04141573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:27.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.294+0000 7f0412d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0414150d10 0x7f04141573b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45094/0 (socket says 172.21.15.78:45094) 2023-10-14T11:58:27.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.294+0000 7f0412d9d700 1 -- 172.21.15.78:0/1935070572 learned_addr learned my addr 172.21.15.78:0/1935070572 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:27.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.294+0000 7f0411d9b700 1 --2- 172.21.15.78:0/1935070572 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0414158860 0x7f04141507d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:27.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.294+0000 7f041259c700 1 --2- 172.21.15.78:0/1935070572 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f03fc003b90 0x7f04141ea2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:27.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.294+0000 7f0412d9d700 1 -- 172.21.15.78:0/1935070572 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f03fc003b90 msgr2=0x7f04141ea2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:27.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.294+0000 7f0412d9d700 1 --2- 172.21.15.78:0/1935070572 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f03fc003b90 0x7f04141ea2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:27.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.294+0000 7f0412d9d700 1 -- 172.21.15.78:0/1935070572 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0414158860 msgr2=0x7f04141507d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:27.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.294+0000 7f0412d9d700 1 --2- 172.21.15.78:0/1935070572 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0414158860 0x7f04141507d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:27.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.294+0000 7f0412d9d700 1 -- 172.21.15.78:0/1935070572 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0414156540 con 0x7f0414150d10 2023-10-14T11:58:27.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.294+0000 7f0412d9d700 1 --2- 172.21.15.78:0/1935070572 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0414150d10 0x7f04141573b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f040c001330 tx=0x7f040c00e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:27.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.294+0000 7f0402ffd700 1 -- 172.21.15.78:0/1935070572 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040c00ebc0 con 0x7f0414150d10 2023-10-14T11:58:27.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.294+0000 7f0402ffd700 1 -- 172.21.15.78:0/1935070572 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f040c00ed20 con 0x7f0414150d10 2023-10-14T11:58:27.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.294+0000 7f0402ffd700 1 -- 172.21.15.78:0/1935070572 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040c013660 con 0x7f0414150d10 2023-10-14T11:58:27.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.294+0000 7f0418a06700 1 -- 172.21.15.78:0/1935070572 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0414156820 con 0x7f0414150d10 2023-10-14T11:58:27.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.294+0000 7f0418a06700 1 -- 172.21.15.78:0/1935070572 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0414156e90 con 0x7f0414150d10 2023-10-14T11:58:27.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.295+0000 7f0418a06700 1 -- 172.21.15.78:0/1935070572 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0414074b80 con 0x7f0414150d10 2023-10-14T11:58:27.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.296+0000 7f0402ffd700 1 -- 172.21.15.78:0/1935070572 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f040c0137c0 con 0x7f0414150d10 2023-10-14T11:58:27.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.296+0000 7f0402ffd700 1 --2- 172.21.15.78:0/1935070572 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f03f406af00 0x7f03f406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:27.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.296+0000 7f0402ffd700 1 -- 172.21.15.78:0/1935070572 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f040c089900 con 0x7f0414150d10 2023-10-14T11:58:27.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.296+0000 7f041259c700 1 --2- 172.21.15.78:0/1935070572 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f03f406af00 0x7f03f406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:27.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.296+0000 7f041259c700 1 --2- 172.21.15.78:0/1935070572 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f03f406af00 0x7f03f406d3c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f04040090b0 tx=0x7f0404008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:27.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.301+0000 7f0402ffd700 1 -- 172.21.15.78:0/1935070572 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f040c054a40 con 0x7f0414150d10 2023-10-14T11:58:27.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.489+0000 7f0418a06700 1 -- 172.21.15.78:0/1935070572 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0414074b80 con 0x7f0414150d10 2023-10-14T11:58:27.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.490+0000 7f0402ffd700 1 -- 172.21.15.78:0/1935070572 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f040c058060 con 0x7f0414150d10 2023-10-14T11:58:27.492 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-10-14T11:58:27.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.492+0000 7f0418a06700 1 -- 172.21.15.78:0/1935070572 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f03f406af00 msgr2=0x7f03f406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:27.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.492+0000 7f0418a06700 1 --2- 172.21.15.78:0/1935070572 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f03f406af00 0x7f03f406d3c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f04040090b0 tx=0x7f0404008030 comp rx=0 tx=0).stop 2023-10-14T11:58:27.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.492+0000 7f0418a06700 1 -- 172.21.15.78:0/1935070572 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0414150d10 msgr2=0x7f04141573b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:27.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.492+0000 7f0418a06700 1 --2- 172.21.15.78:0/1935070572 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0414150d10 0x7f04141573b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f040c001330 tx=0x7f040c00e440 comp rx=0 tx=0).stop 2023-10-14T11:58:27.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.492+0000 7f0418a06700 1 -- 172.21.15.78:0/1935070572 shutdown_connections 2023-10-14T11:58:27.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.492+0000 7f0418a06700 1 --2- 172.21.15.78:0/1935070572 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0414150d10 0x7f04141573b0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:27.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.492+0000 7f0418a06700 1 --2- 172.21.15.78:0/1935070572 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f03fc003b90 0x7f04141ea2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:27.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.492+0000 7f0418a06700 1 --2- 172.21.15.78:0/1935070572 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f03f406af00 0x7f03f406d3c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:27.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.492+0000 7f0418a06700 1 --2- 172.21.15.78:0/1935070572 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0414158860 0x7f04141507d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:27.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.492+0000 7f0418a06700 1 -- 172.21.15.78:0/1935070572 >> 172.21.15.78:0/1935070572 conn(0x7f04140c3a50 msgr2=0x7f04140c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:27.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.492+0000 7f0418a06700 1 -- 172.21.15.78:0/1935070572 shutdown_connections 2023-10-14T11:58:27.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.492+0000 7f0418a06700 1 -- 172.21.15.78:0/1935070572 wait complete. 2023-10-14T11:58:27.506 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T11:58:27.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.627+0000 7fadd9da3700 1 Processor -- start 2023-10-14T11:58:27.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.627+0000 7fadd9da3700 1 -- start start 2023-10-14T11:58:27.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.627+0000 7fadd9da3700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fadd413dbe0 0x7fadd406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:27.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.627+0000 7fadd9da3700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadd406ebe0 con 0x7fadd40c4540 2023-10-14T11:58:27.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.627+0000 7fadd9da3700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadd41558c0 con 0x7fadd406ae10 2023-10-14T11:58:27.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.627+0000 7fadd9da3700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadd406d330 con 0x7fadd413dbe0 2023-10-14T11:58:27.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.627+0000 7fadd37fe700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fadd413dbe0 0x7fadd406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:27.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.627+0000 7fadcbfff700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fadd40c4540 0x7fadd40c02e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49974/0 (socket says 172.21.15.78:49974) 2023-10-14T11:58:27.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.628+0000 7fadcbfff700 1 -- 172.21.15.78:0/3611488306 learned_addr learned my addr 172.21.15.78:0/3611488306 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:27.633 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.628+0000 7fadd37fe700 1 -- 172.21.15.78:0/3611488306 >> v1:172.21.15.195:6789/0 conn(0x7fadd406ae10 legacy=0x7fadd415f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:27.633 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.628+0000 7fadd37fe700 1 -- 172.21.15.78:0/3611488306 >> v1:172.21.15.78:6789/0 conn(0x7fadd40c4540 legacy=0x7fadd40c02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:27.633 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.628+0000 7fadd37fe700 1 -- 172.21.15.78:0/3611488306 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd4154b00 con 0x7fadd413dbe0 2023-10-14T11:58:27.633 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.628+0000 7fadd2ffd700 1 -- 172.21.15.78:0/3611488306 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2392914433 0 0) 0x7fadd406ebe0 con 0x7fadd40c4540 2023-10-14T11:58:27.633 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.628+0000 7fadd37fe700 1 --2- 172.21.15.78:0/3611488306 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadd413dbe0 0x7fadd406a8d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fadc4009840 tx=0x7fadc4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42d5984a9ead7030 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:27.633 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.628+0000 7fadd2ffd700 1 -- 172.21.15.78:0/3611488306 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc4011830 con 0x7fadd413dbe0 2023-10-14T11:58:27.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.628+0000 7fadd2ffd700 1 -- 172.21.15.78:0/3611488306 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadc4011990 con 0x7fadd413dbe0 2023-10-14T11:58:27.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.628+0000 7fadd2ffd700 1 -- 172.21.15.78:0/3611488306 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc4011d20 con 0x7fadd413dbe0 2023-10-14T11:58:27.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.629+0000 7fadd9da3700 1 -- 172.21.15.78:0/3611488306 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadd413dbe0 msgr2=0x7fadd406a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:27.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.629+0000 7fadd9da3700 1 --2- 172.21.15.78:0/3611488306 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadd413dbe0 0x7fadd406a8d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fadc4009840 tx=0x7fadc4009b50 comp rx=0 tx=0).stop 2023-10-14T11:58:27.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.629+0000 7fadd9da3700 1 -- 172.21.15.78:0/3611488306 shutdown_connections 2023-10-14T11:58:27.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.629+0000 7fadd9da3700 1 --2- 172.21.15.78:0/3611488306 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadd413dbe0 0x7fadd406a8d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:27.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.629+0000 7fadd9da3700 1 -- 172.21.15.78:0/3611488306 >> 172.21.15.78:0/3611488306 conn(0x7fadd40c3a50 msgr2=0x7fadd40c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:27.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.629+0000 7fadd9da3700 1 -- 172.21.15.78:0/3611488306 shutdown_connections 2023-10-14T11:58:27.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.629+0000 7fadd9da3700 1 -- 172.21.15.78:0/3611488306 wait complete. 2023-10-14T11:58:27.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.630+0000 7fadd9da3700 1 Processor -- start 2023-10-14T11:58:27.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.630+0000 7fadd9da3700 1 -- start start 2023-10-14T11:58:27.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.630+0000 7fadd9da3700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fadd406ae10 0x7fadd41589e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:27.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.630+0000 7fadd9da3700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fadd40c4540 0x7fadd40c21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:27.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.630+0000 7fadd9da3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadd413dbe0 0x7fadd41ed610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:27.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.630+0000 7fadd9da3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd4156770 con 0x7fadd413dbe0 2023-10-14T11:58:27.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.630+0000 7fadd9da3700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd4157bd0 con 0x7fadd406ae10 2023-10-14T11:58:27.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.630+0000 7fadd9da3700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd41374d0 con 0x7fadd40c4540 2023-10-14T11:58:27.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.631+0000 7fadd3fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadd413dbe0 0x7fadd41ed610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:27.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.631+0000 7fadd3fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadd413dbe0 0x7fadd41ed610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45108/0 (socket says 172.21.15.78:45108) 2023-10-14T11:58:27.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.631+0000 7fadd3fff700 1 -- 172.21.15.78:0/2080996696 learned_addr learned my addr 172.21.15.78:0/2080996696 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:27.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.631+0000 7fadd37fe700 1 --2- 172.21.15.78:0/2080996696 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fadd406ae10 0x7fadd41589e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:27.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.631+0000 7fadcbfff700 1 --2- 172.21.15.78:0/2080996696 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fadd40c4540 0x7fadd40c21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:27.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.631+0000 7fadd3fff700 1 -- 172.21.15.78:0/2080996696 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fadd40c4540 msgr2=0x7fadd40c21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:27.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.631+0000 7fadd3fff700 1 --2- 172.21.15.78:0/2080996696 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fadd40c4540 0x7fadd40c21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:27.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.631+0000 7fadd3fff700 1 -- 172.21.15.78:0/2080996696 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fadd406ae10 msgr2=0x7fadd41589e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:27.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.631+0000 7fadd3fff700 1 --2- 172.21.15.78:0/2080996696 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fadd406ae10 0x7fadd41589e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:27.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.631+0000 7fadd3fff700 1 -- 172.21.15.78:0/2080996696 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc40094a0 con 0x7fadd413dbe0 2023-10-14T11:58:27.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.631+0000 7fadd37fe700 1 --2- 172.21.15.78:0/2080996696 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fadd406ae10 0x7fadd41589e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:27.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.631+0000 7fadd3fff700 1 --2- 172.21.15.78:0/2080996696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadd413dbe0 0x7fadd41ed610 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fadcc00cad0 tx=0x7fadcc00ce90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:27.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.631+0000 7fadd0ff9700 1 -- 172.21.15.78:0/2080996696 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadcc003c00 con 0x7fadd413dbe0 2023-10-14T11:58:27.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.631+0000 7fadd0ff9700 1 -- 172.21.15.78:0/2080996696 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadcc003d60 con 0x7fadd413dbe0 2023-10-14T11:58:27.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.631+0000 7fadd9da3700 1 -- 172.21.15.78:0/2080996696 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadd41ecfe0 con 0x7fadd413dbe0 2023-10-14T11:58:27.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.631+0000 7fadd0ff9700 1 -- 172.21.15.78:0/2080996696 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadcc00ab00 con 0x7fadd413dbe0 2023-10-14T11:58:27.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.631+0000 7fadd9da3700 1 -- 172.21.15.78:0/2080996696 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadd4156490 con 0x7fadd413dbe0 2023-10-14T11:58:27.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.632+0000 7fadd9da3700 1 -- 172.21.15.78:0/2080996696 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadd4137680 con 0x7fadd413dbe0 2023-10-14T11:58:27.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.632+0000 7fadd0ff9700 1 -- 172.21.15.78:0/2080996696 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fadcc00ac60 con 0x7fadd413dbe0 2023-10-14T11:58:27.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.633+0000 7fadd0ff9700 1 --2- 172.21.15.78:0/2080996696 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fadb406af00 0x7fadb406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:27.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.633+0000 7fadd37fe700 1 --2- 172.21.15.78:0/2080996696 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fadb406af00 0x7fadb406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:27.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.633+0000 7fadd0ff9700 1 -- 172.21.15.78:0/2080996696 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fadcc089540 con 0x7fadd413dbe0 2023-10-14T11:58:27.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.633+0000 7fadd37fe700 1 --2- 172.21.15.78:0/2080996696 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fadb406af00 0x7fadb406d3c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fadc4000c00 tx=0x7fadc401c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:27.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.637+0000 7fadd0ff9700 1 -- 172.21.15.78:0/2080996696 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fadd4137680 con 0x7fadd413dbe0 2023-10-14T11:58:27.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.826+0000 7fadd9da3700 1 -- 172.21.15.78:0/2080996696 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fadd4137680 con 0x7fadd413dbe0 2023-10-14T11:58:27.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.826+0000 7fadd0ff9700 1 -- 172.21.15.78:0/2080996696 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fadd4137680 con 0x7fadd413dbe0 2023-10-14T11:58:27.829 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2023-10-14T11:58:27.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.828+0000 7fadd9da3700 1 -- 172.21.15.78:0/2080996696 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fadb406af00 msgr2=0x7fadb406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:27.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.828+0000 7fadd9da3700 1 --2- 172.21.15.78:0/2080996696 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fadb406af00 0x7fadb406d3c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fadc4000c00 tx=0x7fadc401c040 comp rx=0 tx=0).stop 2023-10-14T11:58:27.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.828+0000 7fadd9da3700 1 -- 172.21.15.78:0/2080996696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadd413dbe0 msgr2=0x7fadd41ed610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:27.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.828+0000 7fadd9da3700 1 --2- 172.21.15.78:0/2080996696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadd413dbe0 0x7fadd41ed610 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fadcc00cad0 tx=0x7fadcc00ce90 comp rx=0 tx=0).stop 2023-10-14T11:58:27.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.828+0000 7fadd9da3700 1 -- 172.21.15.78:0/2080996696 shutdown_connections 2023-10-14T11:58:27.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.828+0000 7fadd9da3700 1 --2- 172.21.15.78:0/2080996696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadd413dbe0 0x7fadd41ed610 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:27.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.828+0000 7fadd9da3700 1 --2- 172.21.15.78:0/2080996696 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fadb406af00 0x7fadb406d3c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:27.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.828+0000 7fadd9da3700 1 --2- 172.21.15.78:0/2080996696 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fadd406ae10 0x7fadd41589e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:27.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.828+0000 7fadd9da3700 1 --2- 172.21.15.78:0/2080996696 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fadd40c4540 0x7fadd40c21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:27.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.828+0000 7fadd9da3700 1 -- 172.21.15.78:0/2080996696 >> 172.21.15.78:0/2080996696 conn(0x7fadd40c3a50 msgr2=0x7fadd4071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:27.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.829+0000 7fadd9da3700 1 -- 172.21.15.78:0/2080996696 shutdown_connections 2023-10-14T11:58:27.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.829+0000 7fadd9da3700 1 -- 172.21.15.78:0/2080996696 wait complete. 2023-10-14T11:58:27.842 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T11:58:27.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.960+0000 7fade484d700 1 Processor -- start 2023-10-14T11:58:27.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.960+0000 7fade484d700 1 -- start start 2023-10-14T11:58:27.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.961+0000 7fade484d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fade006ab40 0x7fade006f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:27.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.961+0000 7fade484d700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade006b630 con 0x7fade01554e0 2023-10-14T11:58:27.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.961+0000 7fade484d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade00c49f0 con 0x7fade00c0350 2023-10-14T11:58:27.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.961+0000 7fade484d700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fade00726a0 con 0x7fade006ab40 2023-10-14T11:58:27.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.961+0000 7fadddd9b700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fade01554e0 0x7fade006af20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49990/0 (socket says 172.21.15.78:49990) 2023-10-14T11:58:27.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.961+0000 7fadddd9b700 1 -- 172.21.15.78:0/3585940320 learned_addr learned my addr 172.21.15.78:0/3585940320 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:27.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.961+0000 7fadde59c700 1 --2- 172.21.15.78:0/3585940320 >> v2:172.21.15.195:3300/0 conn(0x7fade006ab40 0x7fade006f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:27.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.961+0000 7faddd59a700 1 -- 172.21.15.78:0/3585940320 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1663469758 0 0) 0x7fade006b630 con 0x7fade01554e0 2023-10-14T11:58:27.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.961+0000 7faddd59a700 1 -- 172.21.15.78:0/3585940320 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadc80035b0 con 0x7fade01554e0 2023-10-14T11:58:27.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.961+0000 7faddd59a700 1 -- 172.21.15.78:0/3585940320 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1351867611 0 0) 0x7fadc80035b0 con 0x7fade01554e0 2023-10-14T11:58:27.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.961+0000 7fadde59c700 1 -- 172.21.15.78:0/3585940320 >> v1:172.21.15.195:6789/0 conn(0x7fade00c0350 legacy=0x7fade013d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:27.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.961+0000 7fadde59c700 1 -- 172.21.15.78:0/3585940320 >> v1:172.21.15.78:6789/0 conn(0x7fade01554e0 legacy=0x7fade006af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:27.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.961+0000 7fadde59c700 1 -- 172.21.15.78:0/3585940320 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade0072980 con 0x7fade006ab40 2023-10-14T11:58:27.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.962+0000 7fadde59c700 1 --2- 172.21.15.78:0/3585940320 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fade006ab40 0x7fade006f1e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fadd4009840 tx=0x7fadd4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c6b474adc7fc48c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:27.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.962+0000 7faddd59a700 1 -- 172.21.15.78:0/3585940320 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd0001670 con 0x7fade006ab40 2023-10-14T11:58:27.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.962+0000 7faddd59a700 1 -- 172.21.15.78:0/3585940320 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadd4004030 con 0x7fade006ab40 2023-10-14T11:58:27.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.962+0000 7faddd59a700 1 -- 172.21.15.78:0/3585940320 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd40043c0 con 0x7fade006ab40 2023-10-14T11:58:27.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.962+0000 7fade484d700 1 -- 172.21.15.78:0/3585940320 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fade006ab40 msgr2=0x7fade006f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:27.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.962+0000 7fade484d700 1 --2- 172.21.15.78:0/3585940320 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fade006ab40 0x7fade006f1e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fadd4009840 tx=0x7fadd4009b50 comp rx=0 tx=0).stop 2023-10-14T11:58:27.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.962+0000 7fade484d700 1 -- 172.21.15.78:0/3585940320 shutdown_connections 2023-10-14T11:58:27.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.962+0000 7fade484d700 1 --2- 172.21.15.78:0/3585940320 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fade006ab40 0x7fade006f1e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:27.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.962+0000 7fade484d700 1 -- 172.21.15.78:0/3585940320 >> 172.21.15.78:0/3585940320 conn(0x7fade00c3ac0 msgr2=0x7fade00c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:27.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.962+0000 7fade484d700 1 -- 172.21.15.78:0/3585940320 shutdown_connections 2023-10-14T11:58:27.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.962+0000 7fade484d700 1 -- 172.21.15.78:0/3585940320 wait complete. 2023-10-14T11:58:27.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.964+0000 7fade484d700 1 Processor -- start 2023-10-14T11:58:27.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.964+0000 7fade484d700 1 -- start start 2023-10-14T11:58:27.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.964+0000 7fade484d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fade006ab40 0x7fade00c2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:27.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.964+0000 7fade484d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fade00c0350 0x7fade01ece90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:27.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.964+0000 7fade484d700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fade01554e0 0x7fade0134740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:27.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.964+0000 7fade484d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade01e98e0 con 0x7fade00c0350 2023-10-14T11:58:27.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.964+0000 7fade484d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade01ea120 con 0x7fade006ab40 2023-10-14T11:58:27.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.964+0000 7fade484d700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fade0157ce0 con 0x7fade01554e0 2023-10-14T11:58:27.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.964+0000 7fadddd9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fade00c0350 0x7fade01ece90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:27.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.965+0000 7fadddd9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fade00c0350 0x7fade01ece90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45118/0 (socket says 172.21.15.78:45118) 2023-10-14T11:58:27.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.965+0000 7fadddd9b700 1 -- 172.21.15.78:0/897869833 learned_addr learned my addr 172.21.15.78:0/897869833 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:27.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.965+0000 7fadde59c700 1 --2- 172.21.15.78:0/897869833 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fade006ab40 0x7fade00c2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:27.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.965+0000 7fadded9d700 1 --2- 172.21.15.78:0/897869833 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fade01554e0 0x7fade0134740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:27.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.965+0000 7fadddd9b700 1 -- 172.21.15.78:0/897869833 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fade01554e0 msgr2=0x7fade0134740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:27.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.965+0000 7fadddd9b700 1 --2- 172.21.15.78:0/897869833 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fade01554e0 0x7fade0134740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:27.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.965+0000 7fadddd9b700 1 -- 172.21.15.78:0/897869833 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fade006ab40 msgr2=0x7fade00c2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:27.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.965+0000 7fadddd9b700 1 --2- 172.21.15.78:0/897869833 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fade006ab40 0x7fade00c2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:27.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.965+0000 7fadddd9b700 1 -- 172.21.15.78:0/897869833 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd40094a0 con 0x7fade00c0350 2023-10-14T11:58:27.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.965+0000 7fadddd9b700 1 --2- 172.21.15.78:0/897869833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fade00c0350 0x7fade01ece90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fadd00011e0 tx=0x7fadd000ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:27.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.965+0000 7fadceffd700 1 -- 172.21.15.78:0/897869833 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd0004c80 con 0x7fade00c0350 2023-10-14T11:58:27.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.965+0000 7fadceffd700 1 -- 172.21.15.78:0/897869833 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadd0004de0 con 0x7fade00c0350 2023-10-14T11:58:27.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.965+0000 7fadceffd700 1 -- 172.21.15.78:0/897869833 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd00072d0 con 0x7fade00c0350 2023-10-14T11:58:27.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.965+0000 7fade484d700 1 -- 172.21.15.78:0/897869833 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fade01e9ad0 con 0x7fade00c0350 2023-10-14T11:58:27.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.965+0000 7fade484d700 1 -- 172.21.15.78:0/897869833 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fade00c31f0 con 0x7fade00c0350 2023-10-14T11:58:27.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.966+0000 7fade484d700 1 -- 172.21.15.78:0/897869833 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fade01ea2d0 con 0x7fade00c0350 2023-10-14T11:58:27.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.967+0000 7fadceffd700 1 -- 172.21.15.78:0/897869833 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fadd0007430 con 0x7fade00c0350 2023-10-14T11:58:27.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.967+0000 7fadceffd700 1 --2- 172.21.15.78:0/897869833 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fadc006af00 0x7fadc006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:27.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.967+0000 7fadceffd700 1 -- 172.21.15.78:0/897869833 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fadd0089310 con 0x7fade00c0350 2023-10-14T11:58:27.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.967+0000 7fadde59c700 1 --2- 172.21.15.78:0/897869833 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fadc006af00 0x7fadc006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:27.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.967+0000 7fadde59c700 1 --2- 172.21.15.78:0/897869833 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fadc006af00 0x7fadc006d3c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fadd40090b0 tx=0x7fadd4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:27.978 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:27.971+0000 7fadceffd700 1 -- 172.21.15.78:0/897869833 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fade01ea2d0 con 0x7fade00c0350 2023-10-14T11:58:28.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.206+0000 7fade484d700 1 -- 172.21.15.78:0/897869833 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fade01ea290 con 0x7fade00c0350 2023-10-14T11:58:28.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.207+0000 7fadceffd700 1 -- 172.21.15.78:0/897869833 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4904 (secure 0 0 0) 0x7fadd0054450 con 0x7fade00c0350 2023-10-14T11:58:28.209 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:28.210 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/353043244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":353043244},{"type":"v1","addr":"172.21.15.78:6836","nonce":353043244}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6834/3389817343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":3389817343},{"type":"v1","addr":"172.21.15.195:6834","nonce":3389817343}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4267,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/460176701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6833","nonce":460176701},{"type":"v1","addr":"172.21.15.195:6835","nonce":460176701}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T11:58:14.535101+0000","modified":"2023-10-14T11:58:17.559202+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/920300787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":920300787},{"type":"v1","addr":"172.21.15.78:6837","nonce":920300787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-14T11:58:28.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.209+0000 7fade484d700 1 -- 172.21.15.78:0/897869833 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fadc006af00 msgr2=0x7fadc006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:28.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.209+0000 7fade484d700 1 --2- 172.21.15.78:0/897869833 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fadc006af00 0x7fadc006d3c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fadd40090b0 tx=0x7fadd4008030 comp rx=0 tx=0).stop 2023-10-14T11:58:28.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.209+0000 7fade484d700 1 -- 172.21.15.78:0/897869833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fade00c0350 msgr2=0x7fade01ece90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:28.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.209+0000 7fade484d700 1 --2- 172.21.15.78:0/897869833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fade00c0350 0x7fade01ece90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fadd00011e0 tx=0x7fadd000ca50 comp rx=0 tx=0).stop 2023-10-14T11:58:28.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.209+0000 7fade484d700 1 -- 172.21.15.78:0/897869833 shutdown_connections 2023-10-14T11:58:28.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.209+0000 7fade484d700 1 --2- 172.21.15.78:0/897869833 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fade01554e0 0x7fade0134740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:28.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.209+0000 7fade484d700 1 --2- 172.21.15.78:0/897869833 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fadc006af00 0x7fadc006d3c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:28.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.209+0000 7fade484d700 1 --2- 172.21.15.78:0/897869833 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fade006ab40 0x7fade00c2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:28.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.209+0000 7fade484d700 1 --2- 172.21.15.78:0/897869833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fade00c0350 0x7fade01ece90 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:28.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.209+0000 7fade484d700 1 -- 172.21.15.78:0/897869833 >> 172.21.15.78:0/897869833 conn(0x7fade00c3ac0 msgr2=0x7fade0153640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:28.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.209+0000 7fade484d700 1 -- 172.21.15.78:0/897869833 shutdown_connections 2023-10-14T11:58:28.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.209+0000 7fade484d700 1 -- 172.21.15.78:0/897869833 wait complete. 2023-10-14T11:58:28.214 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 8 2023-10-14T11:58:28.223 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T11:58:28.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.341+0000 7f3c6dee5700 1 Processor -- start 2023-10-14T11:58:28.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.341+0000 7f3c6dee5700 1 -- start start 2023-10-14T11:58:28.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.342+0000 7f3c6dee5700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c6806f1e0 0x7f3c681554e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:28.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.342+0000 7f3c6dee5700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3c680c0350 0x7f3c6813d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:28.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.342+0000 7f3c6dee5700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c6806b630 con 0x7f3c6806ab40 2023-10-14T11:58:28.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.342+0000 7f3c6dee5700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c680c49f0 con 0x7f3c680c0350 2023-10-14T11:58:28.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.342+0000 7f3c6dee5700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c680726a0 con 0x7f3c6806f1e0 2023-10-14T11:58:28.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.342+0000 7f3c67fff700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3c680c0350 0x7f3c6813d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:28.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.342+0000 7f3c67fff700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3c680c0350 0x7f3c6813d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45126/0 (socket says 172.21.15.78:45126) 2023-10-14T11:58:28.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.342+0000 7f3c67fff700 1 -- 172.21.15.78:0/2292908929 learned_addr learned my addr 172.21.15.78:0/2292908929 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:28.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.342+0000 7f3c66ffd700 1 --2- 172.21.15.78:0/2292908929 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c6806f1e0 0x7f3c681554e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:28.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.342+0000 7f3c67fff700 1 -- 172.21.15.78:0/2292908929 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c6806f1e0 msgr2=0x7f3c681554e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:28.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.342+0000 7f3c67fff700 1 --2- 172.21.15.78:0/2292908929 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c6806f1e0 0x7f3c681554e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:28.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.342+0000 7f3c67fff700 1 -- 172.21.15.78:0/2292908929 >> v1:172.21.15.195:6789/0 conn(0x7f3c6806ab40 legacy=0x7f3c6806af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:28.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.342+0000 7f3c67fff700 1 -- 172.21.15.78:0/2292908929 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c68072980 con 0x7f3c680c0350 2023-10-14T11:58:28.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.342+0000 7f3c67fff700 1 --2- 172.21.15.78:0/2292908929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c680c0350 0x7f3c6813d7b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3c60009e50 tx=0x7f3c6000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b5dae439942f272 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:28.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.343+0000 7f3c667fc700 1 -- 172.21.15.78:0/2292908929 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6000e070 con 0x7f3c680c0350 2023-10-14T11:58:28.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.343+0000 7f3c667fc700 1 -- 172.21.15.78:0/2292908929 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c6000b9c0 con 0x7f3c680c0350 2023-10-14T11:58:28.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.343+0000 7f3c667fc700 1 -- 172.21.15.78:0/2292908929 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6000bd70 con 0x7f3c680c0350 2023-10-14T11:58:28.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.343+0000 7f3c6dee5700 1 -- 172.21.15.78:0/2292908929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c680c0350 msgr2=0x7f3c6813d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:28.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.343+0000 7f3c6dee5700 1 --2- 172.21.15.78:0/2292908929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c680c0350 0x7f3c6813d7b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3c60009e50 tx=0x7f3c6000b330 comp rx=0 tx=0).stop 2023-10-14T11:58:28.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.343+0000 7f3c6dee5700 1 -- 172.21.15.78:0/2292908929 shutdown_connections 2023-10-14T11:58:28.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.343+0000 7f3c6dee5700 1 --2- 172.21.15.78:0/2292908929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c680c0350 0x7f3c6813d7b0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:28.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.343+0000 7f3c6dee5700 1 --2- 172.21.15.78:0/2292908929 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c6806f1e0 0x7f3c681554e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:28.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.343+0000 7f3c6dee5700 1 -- 172.21.15.78:0/2292908929 >> 172.21.15.78:0/2292908929 conn(0x7f3c680c3ac0 msgr2=0x7f3c680c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:28.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.343+0000 7f3c6dee5700 1 -- 172.21.15.78:0/2292908929 shutdown_connections 2023-10-14T11:58:28.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.343+0000 7f3c6dee5700 1 -- 172.21.15.78:0/2292908929 wait complete. 2023-10-14T11:58:28.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.344+0000 7f3c6dee5700 1 Processor -- start 2023-10-14T11:58:28.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.344+0000 7f3c6dee5700 1 -- start start 2023-10-14T11:58:28.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.345+0000 7f3c6dee5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c6806ab40 0x7f3c680c67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:28.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.345+0000 7f3c6dee5700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c6806f1e0 0x7f3c681ea220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:28.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.345+0000 7f3c6dee5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c680c0350 0x7f3c680ab3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:28.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.345+0000 7f3c6dee5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c68151da0 con 0x7f3c680c0350 2023-10-14T11:58:28.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.345+0000 7f3c6dee5700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c681e9af0 con 0x7f3c6806ab40 2023-10-14T11:58:28.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.345+0000 7f3c6dee5700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c681e9d90 con 0x7f3c6806f1e0 2023-10-14T11:58:28.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.345+0000 7f3c67fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c680c0350 0x7f3c680ab3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:28.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.345+0000 7f3c67fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c680c0350 0x7f3c680ab3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45136/0 (socket says 172.21.15.78:45136) 2023-10-14T11:58:28.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.345+0000 7f3c67fff700 1 -- 172.21.15.78:0/1609012917 learned_addr learned my addr 172.21.15.78:0/1609012917 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:28.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.345+0000 7f3c677fe700 1 --2- 172.21.15.78:0/1609012917 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c6806ab40 0x7f3c680c67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:28.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.345+0000 7f3c66ffd700 1 --2- 172.21.15.78:0/1609012917 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c6806f1e0 0x7f3c681ea220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:28.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.345+0000 7f3c67fff700 1 -- 172.21.15.78:0/1609012917 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c6806f1e0 msgr2=0x7f3c681ea220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:28.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.345+0000 7f3c67fff700 1 --2- 172.21.15.78:0/1609012917 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c6806f1e0 0x7f3c681ea220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:28.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.345+0000 7f3c67fff700 1 -- 172.21.15.78:0/1609012917 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c6806ab40 msgr2=0x7f3c680c67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:28.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.345+0000 7f3c67fff700 1 --2- 172.21.15.78:0/1609012917 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c6806ab40 0x7f3c680c67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:28.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.346+0000 7f3c67fff700 1 -- 172.21.15.78:0/1609012917 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c60009b30 con 0x7f3c680c0350 2023-10-14T11:58:28.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.346+0000 7f3c67fff700 1 --2- 172.21.15.78:0/1609012917 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c680c0350 0x7f3c680ab3f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3c680ad290 tx=0x7f3c60004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:28.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.346+0000 7f3c4ffff700 1 -- 172.21.15.78:0/1609012917 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c60004960 con 0x7f3c680c0350 2023-10-14T11:58:28.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.346+0000 7f3c4ffff700 1 -- 172.21.15.78:0/1609012917 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c600099b0 con 0x7f3c680c0350 2023-10-14T11:58:28.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.346+0000 7f3c6dee5700 1 -- 172.21.15.78:0/1609012917 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c681ede10 con 0x7f3c680c0350 2023-10-14T11:58:28.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.346+0000 7f3c4ffff700 1 -- 172.21.15.78:0/1609012917 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6001b670 con 0x7f3c680c0350 2023-10-14T11:58:28.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.346+0000 7f3c6dee5700 1 -- 172.21.15.78:0/1609012917 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c68151ac0 con 0x7f3c680c0350 2023-10-14T11:58:28.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.347+0000 7f3c6dee5700 1 -- 172.21.15.78:0/1609012917 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c681e9f00 con 0x7f3c680c0350 2023-10-14T11:58:28.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.347+0000 7f3c4ffff700 1 -- 172.21.15.78:0/1609012917 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3c6001b7d0 con 0x7f3c680c0350 2023-10-14T11:58:28.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.347+0000 7f3c4ffff700 1 --2- 172.21.15.78:0/1609012917 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f3c5406af00 0x7f3c5406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:28.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.348+0000 7f3c677fe700 1 --2- 172.21.15.78:0/1609012917 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f3c5406af00 0x7f3c5406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:28.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.348+0000 7f3c4ffff700 1 -- 172.21.15.78:0/1609012917 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3c6008aed0 con 0x7f3c680c0350 2023-10-14T11:58:28.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.348+0000 7f3c677fe700 1 --2- 172.21.15.78:0/1609012917 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f3c5406af00 0x7f3c5406d3c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c000f60 tx=0x7f3c5c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:28.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.352+0000 7f3c4ffff700 1 -- 172.21.15.78:0/1609012917 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3c60056010 con 0x7f3c680c0350 2023-10-14T11:58:28.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.543+0000 7f3c6dee5700 1 -- 172.21.15.78:0/1609012917 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c68074b80 con 0x7f3c680c0350 2023-10-14T11:58:28.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.545+0000 7f3c4ffff700 1 -- 172.21.15.78:0/1609012917 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16481 (secure 0 0 0) 0x7f3c60059630 con 0x7f3c680c0350 2023-10-14T11:58:28.547 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:28.547 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":21,"fsid":"4c5dc134-3092-406a-a1e1-13e6d9e926e8","created":"2023-10-14T11:57:57.363750+0000","modified":"2023-10-14T11:58:16.139413+0000","last_up_change":"2023-10-14T11:58:06.982472+0000","last_in_change":"2023-10-14T11:57:59.821373+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T11:58:09.001200+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T11:58:12.778573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T11:58:13.490286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"625a5fc0-9474-4eae-b80e-14c27a74ad19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6801","nonce":3884969780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6803","nonce":3884969780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6807","nonce":3884969780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6805","nonce":3884969780}]},"public_addr":"172.21.15.78:6801/3884969780","cluster_addr":"172.21.15.78:6803/3884969780","heartbeat_back_addr":"172.21.15.78:6807/3884969780","heartbeat_front_addr":"172.21.15.78:6805/3884969780","state":["exists","up"]},{"osd":1,"uuid":"49d63db1-2981-49ab-9c40-c50fc744b982","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6809","nonce":2176928406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6811","nonce":2176928406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6815","nonce":2176928406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6813","nonce":2176928406}]},"public_addr":"172.21.15.78:6809/2176928406","cluster_addr":"172.21.15.78:6811/2176928406","heartbeat_back_addr":"172.21.15.78:6815/2176928406","heartbeat_front_addr":"172.21.15.78:6813/2176928406","state":["exists","up"]},{"osd":2,"uuid":"1d7d0b29-cbd1-4717-9fe6-765a8d4998b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6825","nonce":3647044577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6827","nonce":3647044577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6831","nonce":3647044577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6829","nonce":3647044577}]},"public_addr":"172.21.15.78:6825/3647044577","cluster_addr":"172.21.15.78:6827/3647044577","heartbeat_back_addr":"172.21.15.78:6831/3647044577","heartbeat_front_addr":"172.21.15.78:6829/3647044577","state":["exists","up"]},{"osd":3,"uuid":"2f75ded1-e9e2-40e7-99ae-45e1e425e1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6817","nonce":3584112288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6819","nonce":3584112288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6823","nonce":3584112288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6821","nonce":3584112288}]},"public_addr":"172.21.15.78:6817/3584112288","cluster_addr":"172.21.15.78:6819/3584112288","heartbeat_back_addr":"172.21.15.78:6823/3584112288","heartbeat_front_addr":"172.21.15.78:6821/3584112288","state":["exists","up"]},{"osd":4,"uuid":"5130b432-8358-46cb-8a31-3064c976f6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6801","nonce":3732756301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6803","nonce":3732756301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6807","nonce":3732756301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6805","nonce":3732756301}]},"public_addr":"172.21.15.195:6801/3732756301","cluster_addr":"172.21.15.195:6803/3732756301","heartbeat_back_addr":"172.21.15.195:6807/3732756301","heartbeat_front_addr":"172.21.15.195:6805/3732756301","state":["exists","up"]},{"osd":5,"uuid":"60e234a9-a506-4e40-9c19-49a58f3cf28c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6825","nonce":2857706411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6827","nonce":2857706411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6831","nonce":2857706411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6829","nonce":2857706411}]},"public_addr":"172.21.15.195:6825/2857706411","cluster_addr":"172.21.15.195:6827/2857706411","heartbeat_back_addr":"172.21.15.195:6831/2857706411","heartbeat_front_addr":"172.21.15.195:6829/2857706411","state":["exists","up"]},{"osd":6,"uuid":"df7d2dd6-26b6-4eb1-96e5-94d1f3955b33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6810","nonce":72554374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6813","nonce":72554374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6820","nonce":72554374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6817","nonce":72554374}]},"public_addr":"172.21.15.195:6810/72554374","cluster_addr":"172.21.15.195:6813/72554374","heartbeat_back_addr":"172.21.15.195:6820/72554374","heartbeat_front_addr":"172.21.15.195:6817/72554374","state":["exists","up"]},{"osd":7,"uuid":"47a9d4de-f56d-4aa4-9900-c4a9cf9b2f97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6809","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6811","nonce":482472979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6816","nonce":482472979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6823","nonce":482472979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6821","nonce":482472979}]},"public_addr":"172.21.15.195:6811/482472979","cluster_addr":"172.21.15.195:6816/482472979","heartbeat_back_addr":"172.21.15.195:6823/482472979","heartbeat_front_addr":"172.21.15.195:6821/482472979","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.754003+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.005994+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.043603+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.024506+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.062585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.033767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.994825+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.038300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T11:58:28.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.546+0000 7f3c6dee5700 1 -- 172.21.15.78:0/1609012917 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f3c5406af00 msgr2=0x7f3c5406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:28.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.546+0000 7f3c6dee5700 1 --2- 172.21.15.78:0/1609012917 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f3c5406af00 0x7f3c5406d3c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c000f60 tx=0x7f3c5c009040 comp rx=0 tx=0).stop 2023-10-14T11:58:28.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.546+0000 7f3c6dee5700 1 -- 172.21.15.78:0/1609012917 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c680c0350 msgr2=0x7f3c680ab3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:28.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.546+0000 7f3c6dee5700 1 --2- 172.21.15.78:0/1609012917 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c680c0350 0x7f3c680ab3f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3c680ad290 tx=0x7f3c60004ad0 comp rx=0 tx=0).stop 2023-10-14T11:58:28.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.547+0000 7f3c6dee5700 1 -- 172.21.15.78:0/1609012917 shutdown_connections 2023-10-14T11:58:28.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.547+0000 7f3c6dee5700 1 --2- 172.21.15.78:0/1609012917 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c680c0350 0x7f3c680ab3f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:28.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.547+0000 7f3c6dee5700 1 --2- 172.21.15.78:0/1609012917 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f3c5406af00 0x7f3c5406d3c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:28.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.547+0000 7f3c6dee5700 1 --2- 172.21.15.78:0/1609012917 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c6806ab40 0x7f3c680c67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:28.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.547+0000 7f3c6dee5700 1 --2- 172.21.15.78:0/1609012917 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c6806f1e0 0x7f3c681ea220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:28.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.547+0000 7f3c6dee5700 1 -- 172.21.15.78:0/1609012917 >> 172.21.15.78:0/1609012917 conn(0x7f3c680c3ac0 msgr2=0x7f3c680667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:28.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.547+0000 7f3c6dee5700 1 -- 172.21.15.78:0/1609012917 shutdown_connections 2023-10-14T11:58:28.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.547+0000 7f3c6dee5700 1 -- 172.21.15.78:0/1609012917 wait complete. 2023-10-14T11:58:28.561 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T11:58:28.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.678+0000 7fa2d899f700 1 Processor -- start 2023-10-14T11:58:28.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.678+0000 7fa2d899f700 1 -- start start 2023-10-14T11:58:28.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.679+0000 7fa2d899f700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa2d406ab40 0x7fa2d406f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:28.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.679+0000 7fa2d899f700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa2d40c0350 0x7fa2d413d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:28.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.679+0000 7fa2d899f700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2d406b630 con 0x7fa2d41554e0 2023-10-14T11:58:28.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.679+0000 7fa2d899f700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2d40c49f0 con 0x7fa2d406ab40 2023-10-14T11:58:28.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.679+0000 7fa2d899f700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d40726a0 con 0x7fa2d40c0350 2023-10-14T11:58:28.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.679+0000 7fa2d259c700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa2d406ab40 0x7fa2d406f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:28.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.679+0000 7fa2d259c700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa2d406ab40 0x7fa2d406f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45140/0 (socket says 172.21.15.78:45140) 2023-10-14T11:58:28.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.679+0000 7fa2d259c700 1 -- 172.21.15.78:0/3992758708 learned_addr learned my addr 172.21.15.78:0/3992758708 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:28.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.680+0000 7fa2d2d9d700 1 --2- 172.21.15.78:0/3992758708 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa2d40c0350 0x7fa2d413d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:28.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.680+0000 7fa2d259c700 1 -- 172.21.15.78:0/3992758708 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa2d40c0350 msgr2=0x7fa2d413d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:28.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.680+0000 7fa2d259c700 1 --2- 172.21.15.78:0/3992758708 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa2d40c0350 0x7fa2d413d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:28.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.680+0000 7fa2d259c700 1 -- 172.21.15.78:0/3992758708 >> v1:172.21.15.195:6789/0 conn(0x7fa2d41554e0 legacy=0x7fa2d406af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:28.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.680+0000 7fa2d259c700 1 -- 172.21.15.78:0/3992758708 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d4072980 con 0x7fa2d406ab40 2023-10-14T11:58:28.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.680+0000 7fa2d259c700 1 --2- 172.21.15.78:0/3992758708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2d406ab40 0x7fa2d406f1e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa2c80098f0 tx=0x7fa2c8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f91e1695e8451b07 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:28.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.680+0000 7fa2d159a700 1 -- 172.21.15.78:0/3992758708 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c800b8a0 con 0x7fa2d406ab40 2023-10-14T11:58:28.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.680+0000 7fa2d159a700 1 -- 172.21.15.78:0/3992758708 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2c800ba00 con 0x7fa2d406ab40 2023-10-14T11:58:28.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.680+0000 7fa2d159a700 1 -- 172.21.15.78:0/3992758708 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c800bd90 con 0x7fa2d406ab40 2023-10-14T11:58:28.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.680+0000 7fa2d899f700 1 -- 172.21.15.78:0/3992758708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2d406ab40 msgr2=0x7fa2d406f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:28.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.680+0000 7fa2d899f700 1 --2- 172.21.15.78:0/3992758708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2d406ab40 0x7fa2d406f1e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa2c80098f0 tx=0x7fa2c8009c00 comp rx=0 tx=0).stop 2023-10-14T11:58:28.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.681+0000 7fa2d899f700 1 -- 172.21.15.78:0/3992758708 shutdown_connections 2023-10-14T11:58:28.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.681+0000 7fa2d899f700 1 --2- 172.21.15.78:0/3992758708 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa2d40c0350 0x7fa2d413d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:28.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.681+0000 7fa2d899f700 1 --2- 172.21.15.78:0/3992758708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2d406ab40 0x7fa2d406f1e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:28.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.681+0000 7fa2d899f700 1 -- 172.21.15.78:0/3992758708 >> 172.21.15.78:0/3992758708 conn(0x7fa2d40c3ac0 msgr2=0x7fa2d40c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:28.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.681+0000 7fa2d899f700 1 -- 172.21.15.78:0/3992758708 shutdown_connections 2023-10-14T11:58:28.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.681+0000 7fa2d899f700 1 -- 172.21.15.78:0/3992758708 wait complete. 2023-10-14T11:58:28.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.682+0000 7fa2d899f700 1 Processor -- start 2023-10-14T11:58:28.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.682+0000 7fa2d899f700 1 -- start start 2023-10-14T11:58:28.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.683+0000 7fa2d899f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa2d406ab40 0x7fa2d41581b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:28.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.683+0000 7fa2d899f700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa2d40c0350 0x7fa2d41eafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:28.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.683+0000 7fa2d899f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2d41554e0 0x7fa2d4138ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:28.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.683+0000 7fa2d899f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d4157d20 con 0x7fa2d41554e0 2023-10-14T11:58:28.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.683+0000 7fa2d899f700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d40a9120 con 0x7fa2d406ab40 2023-10-14T11:58:28.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.683+0000 7fa2d899f700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d414efb0 con 0x7fa2d40c0350 2023-10-14T11:58:28.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.683+0000 7fa2d2d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2d41554e0 0x7fa2d4138ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:28.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.683+0000 7fa2d2d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2d41554e0 0x7fa2d4138ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45150/0 (socket says 172.21.15.78:45150) 2023-10-14T11:58:28.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.683+0000 7fa2d2d9d700 1 -- 172.21.15.78:0/1537871752 learned_addr learned my addr 172.21.15.78:0/1537871752 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:28.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.683+0000 7fa2d259c700 1 --2- 172.21.15.78:0/1537871752 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa2d406ab40 0x7fa2d41581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:28.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.683+0000 7fa2d1d9b700 1 --2- 172.21.15.78:0/1537871752 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa2d40c0350 0x7fa2d41eafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:28.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.683+0000 7fa2d2d9d700 1 -- 172.21.15.78:0/1537871752 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa2d40c0350 msgr2=0x7fa2d41eafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:28.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.683+0000 7fa2d2d9d700 1 --2- 172.21.15.78:0/1537871752 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa2d40c0350 0x7fa2d41eafb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:28.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.683+0000 7fa2d2d9d700 1 -- 172.21.15.78:0/1537871752 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa2d406ab40 msgr2=0x7fa2d41581b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:28.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.683+0000 7fa2d2d9d700 1 --2- 172.21.15.78:0/1537871752 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa2d406ab40 0x7fa2d41581b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:28.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.683+0000 7fa2d2d9d700 1 -- 172.21.15.78:0/1537871752 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c8009550 con 0x7fa2d41554e0 2023-10-14T11:58:28.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.683+0000 7fa2d2d9d700 1 --2- 172.21.15.78:0/1537871752 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2d41554e0 0x7fa2d4138ab0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc007e70 tx=0x7fa2cc00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:28.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.683+0000 7fa2c2ffd700 1 -- 172.21.15.78:0/1537871752 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2cc00ee50 con 0x7fa2d41554e0 2023-10-14T11:58:28.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.683+0000 7fa2c2ffd700 1 -- 172.21.15.78:0/1537871752 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2cc014420 con 0x7fa2d41554e0 2023-10-14T11:58:28.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.683+0000 7fa2d899f700 1 -- 172.21.15.78:0/1537871752 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2d414f3d0 con 0x7fa2d41554e0 2023-10-14T11:58:28.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.683+0000 7fa2c2ffd700 1 -- 172.21.15.78:0/1537871752 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2cc0147d0 con 0x7fa2d41554e0 2023-10-14T11:58:28.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.683+0000 7fa2d899f700 1 -- 172.21.15.78:0/1537871752 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2d41576d0 con 0x7fa2d41554e0 2023-10-14T11:58:28.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.684+0000 7fa2d899f700 1 -- 172.21.15.78:0/1537871752 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2d4157ed0 con 0x7fa2d41554e0 2023-10-14T11:58:28.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.685+0000 7fa2c2ffd700 1 -- 172.21.15.78:0/1537871752 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa2cc014930 con 0x7fa2d41554e0 2023-10-14T11:58:28.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.685+0000 7fa2c2ffd700 1 --2- 172.21.15.78:0/1537871752 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa2b406af00 0x7fa2b406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:28.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.685+0000 7fa2c2ffd700 1 -- 172.21.15.78:0/1537871752 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa2cc089d60 con 0x7fa2d41554e0 2023-10-14T11:58:28.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.685+0000 7fa2d259c700 1 --2- 172.21.15.78:0/1537871752 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa2b406af00 0x7fa2b406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:28.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.685+0000 7fa2d259c700 1 --2- 172.21.15.78:0/1537871752 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa2b406af00 0x7fa2b406d3c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8014040 tx=0x7fa2c8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:28.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.689+0000 7fa2c2ffd700 1 -- 172.21.15.78:0/1537871752 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa2d4157ed0 con 0x7fa2d41554e0 2023-10-14T11:58:28.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.920+0000 7fa2d899f700 1 -- 172.21.15.78:0/1537871752 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2d4157ed0 con 0x7fa2d41554e0 2023-10-14T11:58:28.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.921+0000 7fa2c2ffd700 1 -- 172.21.15.78:0/1537871752 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4904 (secure 0 0 0) 0x7fa2d4157ed0 con 0x7fa2d41554e0 2023-10-14T11:58:28.923 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:28.923 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/353043244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":353043244},{"type":"v1","addr":"172.21.15.78:6836","nonce":353043244}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6834/3389817343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":3389817343},{"type":"v1","addr":"172.21.15.195:6834","nonce":3389817343}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4267,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/460176701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6833","nonce":460176701},{"type":"v1","addr":"172.21.15.195:6835","nonce":460176701}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T11:58:14.535101+0000","modified":"2023-10-14T11:58:17.559202+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/920300787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":920300787},{"type":"v1","addr":"172.21.15.78:6837","nonce":920300787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-14T11:58:28.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.923+0000 7fa2d899f700 1 -- 172.21.15.78:0/1537871752 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa2b406af00 msgr2=0x7fa2b406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:28.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.923+0000 7fa2d899f700 1 --2- 172.21.15.78:0/1537871752 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa2b406af00 0x7fa2b406d3c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8014040 tx=0x7fa2c8008030 comp rx=0 tx=0).stop 2023-10-14T11:58:28.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.923+0000 7fa2d899f700 1 -- 172.21.15.78:0/1537871752 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2d41554e0 msgr2=0x7fa2d4138ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:28.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.923+0000 7fa2d899f700 1 --2- 172.21.15.78:0/1537871752 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2d41554e0 0x7fa2d4138ab0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc007e70 tx=0x7fa2cc00e3c0 comp rx=0 tx=0).stop 2023-10-14T11:58:28.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.923+0000 7fa2d899f700 1 -- 172.21.15.78:0/1537871752 shutdown_connections 2023-10-14T11:58:28.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.923+0000 7fa2d899f700 1 --2- 172.21.15.78:0/1537871752 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2d41554e0 0x7fa2d4138ab0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:28.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.923+0000 7fa2d899f700 1 --2- 172.21.15.78:0/1537871752 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa2b406af00 0x7fa2b406d3c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:28.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.923+0000 7fa2d899f700 1 --2- 172.21.15.78:0/1537871752 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa2d406ab40 0x7fa2d41581b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:28.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.923+0000 7fa2d899f700 1 --2- 172.21.15.78:0/1537871752 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa2d40c0350 0x7fa2d41eafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:28.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.923+0000 7fa2d899f700 1 -- 172.21.15.78:0/1537871752 >> 172.21.15.78:0/1537871752 conn(0x7fa2d40c3ac0 msgr2=0x7fa2d40667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:28.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.923+0000 7fa2d899f700 1 -- 172.21.15.78:0/1537871752 shutdown_connections 2023-10-14T11:58:28.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:28.923+0000 7fa2d899f700 1 -- 172.21.15.78:0/1537871752 wait complete. 2023-10-14T11:58:28.929 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 8 2023-10-14T11:58:29.339 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-14T11:58:14.535101+0000', 'modified': '2023-10-14T11:58:17.559202+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4269}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4269': {'gid': 4269, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.78:6837/920300787', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 920300787}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 920300787}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-14T11:58:29.340 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-14T11:58:29.340 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-14T11:58:29.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.459+0000 7fa5e78e4700 1 Processor -- start 2023-10-14T11:58:29.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.460+0000 7fa5e78e4700 1 -- start start 2023-10-14T11:58:29.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.460+0000 7fa5e78e4700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa5e01500a0 0x7fa5e00a9c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:29.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.460+0000 7fa5e78e4700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa5e00adaa0 0x7fa5e00aa170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:29.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.460+0000 7fa5e78e4700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5e0070260 con 0x7fa5e00ad550 2023-10-14T11:58:29.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.460+0000 7fa5e78e4700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5e0153420 con 0x7fa5e01500a0 2023-10-14T11:58:29.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.460+0000 7fa5e78e4700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5e0070500 con 0x7fa5e00adaa0 2023-10-14T11:58:29.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.460+0000 7fa5e5680700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa5e01500a0 0x7fa5e00a9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:29.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.460+0000 7fa5e4e7f700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fa5e00ad550 0x7fa5e00c3fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50030/0 (socket says 172.21.15.78:50030) 2023-10-14T11:58:29.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.460+0000 7fa5e4e7f700 1 -- 172.21.15.78:0/2889238170 learned_addr learned my addr 172.21.15.78:0/2889238170 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:29.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.461+0000 7fa5e5e81700 1 --2- 172.21.15.78:0/2889238170 >> v2:172.21.15.195:3300/0 conn(0x7fa5e00adaa0 0x7fa5e00aa170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:29.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.461+0000 7fa5e5680700 1 -- 172.21.15.78:0/2889238170 >> v2:172.21.15.195:3300/0 conn(0x7fa5e00adaa0 msgr2=0x7fa5e00aa170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:29.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.461+0000 7fa5e5680700 1 --2- 172.21.15.78:0/2889238170 >> v2:172.21.15.195:3300/0 conn(0x7fa5e00adaa0 0x7fa5e00aa170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.461+0000 7fa5e5680700 1 -- 172.21.15.78:0/2889238170 >> v1:172.21.15.78:6789/0 conn(0x7fa5e00ad550 legacy=0x7fa5e00c3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:29.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.461+0000 7fa5e5680700 1 -- 172.21.15.78:0/2889238170 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e006f770 con 0x7fa5e01500a0 2023-10-14T11:58:29.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.461+0000 7fa5dffff700 1 -- 172.21.15.78:0/2889238170 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4247908230 0 0) 0x7fa5e0070260 con 0x7fa5e00ad550 2023-10-14T11:58:29.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.461+0000 7fa5e5680700 1 --2- 172.21.15.78:0/2889238170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5e01500a0 0x7fa5e00a9c30 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa5e006f3c0 tx=0x7fa5d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f94d0013f75013a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:29.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.461+0000 7fa5dffff700 1 -- 172.21.15.78:0/2889238170 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d000e070 con 0x7fa5e01500a0 2023-10-14T11:58:29.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.461+0000 7fa5dffff700 1 -- 172.21.15.78:0/2889238170 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d000bbd0 con 0x7fa5e01500a0 2023-10-14T11:58:29.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.461+0000 7fa5dffff700 1 -- 172.21.15.78:0/2889238170 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d0012430 con 0x7fa5e01500a0 2023-10-14T11:58:29.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.462+0000 7fa5e78e4700 1 -- 172.21.15.78:0/2889238170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5e01500a0 msgr2=0x7fa5e00a9c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:29.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.462+0000 7fa5e78e4700 1 --2- 172.21.15.78:0/2889238170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5e01500a0 0x7fa5e00a9c30 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa5e006f3c0 tx=0x7fa5d000b330 comp rx=0 tx=0).stop 2023-10-14T11:58:29.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.462+0000 7fa5e78e4700 1 -- 172.21.15.78:0/2889238170 shutdown_connections 2023-10-14T11:58:29.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.462+0000 7fa5e78e4700 1 --2- 172.21.15.78:0/2889238170 >> v2:172.21.15.195:3300/0 conn(0x7fa5e00adaa0 0x7fa5e00aa170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.462+0000 7fa5e78e4700 1 --2- 172.21.15.78:0/2889238170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5e01500a0 0x7fa5e00a9c30 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.462+0000 7fa5e78e4700 1 -- 172.21.15.78:0/2889238170 >> 172.21.15.78:0/2889238170 conn(0x7fa5e0072a00 msgr2=0x7fa5e00733e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:29.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.462+0000 7fa5e78e4700 1 -- 172.21.15.78:0/2889238170 shutdown_connections 2023-10-14T11:58:29.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.462+0000 7fa5e78e4700 1 -- 172.21.15.78:0/2889238170 wait complete. 2023-10-14T11:58:29.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.463+0000 7fa5e78e4700 1 Processor -- start 2023-10-14T11:58:29.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.463+0000 7fa5e78e4700 1 -- start start 2023-10-14T11:58:29.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.463+0000 7fa5e78e4700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa5e00ad550 0x7fa5e00c18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:29.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.463+0000 7fa5e78e4700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5e00adaa0 0x7fa5e01eb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:29.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.463+0000 7fa5e78e4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5e01500a0 0x7fa5e01eb560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:29.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.463+0000 7fa5e78e4700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e00c07d0 con 0x7fa5e01500a0 2023-10-14T11:58:29.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.463+0000 7fa5e78e4700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e01eab90 con 0x7fa5e00adaa0 2023-10-14T11:58:29.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.463+0000 7fa5e78e4700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e01ebaa0 con 0x7fa5e00ad550 2023-10-14T11:58:29.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.463+0000 7fa5e5e81700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5e01500a0 0x7fa5e01eb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:29.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.464+0000 7fa5e5e81700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5e01500a0 0x7fa5e01eb560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45162/0 (socket says 172.21.15.78:45162) 2023-10-14T11:58:29.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.464+0000 7fa5e5e81700 1 -- 172.21.15.78:0/2258725310 learned_addr learned my addr 172.21.15.78:0/2258725310 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:29.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.464+0000 7fa5e4e7f700 1 --2- 172.21.15.78:0/2258725310 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5e00adaa0 0x7fa5e01eb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:29.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.464+0000 7fa5e5680700 1 --2- 172.21.15.78:0/2258725310 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa5e00ad550 0x7fa5e00c18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:29.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.464+0000 7fa5e5e81700 1 -- 172.21.15.78:0/2258725310 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa5e00ad550 msgr2=0x7fa5e00c18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:29.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.464+0000 7fa5e5e81700 1 --2- 172.21.15.78:0/2258725310 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa5e00ad550 0x7fa5e00c18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.464+0000 7fa5e5e81700 1 -- 172.21.15.78:0/2258725310 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5e00adaa0 msgr2=0x7fa5e01eb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:29.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.464+0000 7fa5e5e81700 1 --2- 172.21.15.78:0/2258725310 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5e00adaa0 0x7fa5e01eb020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.464+0000 7fa5e5e81700 1 -- 172.21.15.78:0/2258725310 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d0009a70 con 0x7fa5e01500a0 2023-10-14T11:58:29.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.464+0000 7fa5e5680700 1 --2- 172.21.15.78:0/2258725310 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa5e00ad550 0x7fa5e00c18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:29.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.464+0000 7fa5e5e81700 1 --2- 172.21.15.78:0/2258725310 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5e01500a0 0x7fa5e01eb560 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa5d800e810 tx=0x7fa5d800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:29.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.464+0000 7fa5ddffb700 1 -- 172.21.15.78:0/2258725310 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d800c070 con 0x7fa5e01500a0 2023-10-14T11:58:29.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.464+0000 7fa5ddffb700 1 -- 172.21.15.78:0/2258725310 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d8013650 con 0x7fa5e01500a0 2023-10-14T11:58:29.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.464+0000 7fa5e78e4700 1 -- 172.21.15.78:0/2258725310 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5e01e8a90 con 0x7fa5e01500a0 2023-10-14T11:58:29.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.464+0000 7fa5ddffb700 1 -- 172.21.15.78:0/2258725310 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d8013a00 con 0x7fa5e01500a0 2023-10-14T11:58:29.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.464+0000 7fa5e78e4700 1 -- 172.21.15.78:0/2258725310 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5e01e80c0 con 0x7fa5e01500a0 2023-10-14T11:58:29.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.466+0000 7fa5ddffb700 1 -- 172.21.15.78:0/2258725310 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa5d8013b60 con 0x7fa5e01500a0 2023-10-14T11:58:29.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.466+0000 7fa5ddffb700 1 --2- 172.21.15.78:0/2258725310 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa5cc06af00 0x7fa5cc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:29.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.466+0000 7fa5ddffb700 1 -- 172.21.15.78:0/2258725310 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa5d8089070 con 0x7fa5e01500a0 2023-10-14T11:58:29.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.466+0000 7fa5e5680700 1 --2- 172.21.15.78:0/2258725310 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa5cc06af00 0x7fa5cc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:29.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.466+0000 7fa5e5680700 1 --2- 172.21.15.78:0/2258725310 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa5cc06af00 0x7fa5cc06d3c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0014040 tx=0x7fa5d0011000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:29.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.468+0000 7fa5f3aceb80 1 Processor -- start 2023-10-14T11:58:29.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.468+0000 7fa5f3aceb80 1 -- start start 2023-10-14T11:58:29.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.469+0000 7fa5f3aceb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a71a2f6b0 0x562a71a6d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:29.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.469+0000 7fa5f3aceb80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562a71a2fa90 0x562a71a6db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:29.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.469+0000 7fa5f3aceb80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x562a71a29c90 0x562a71a4d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:29.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.469+0000 7fa5f3aceb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a719a3c20 con 0x562a71a2f6b0 2023-10-14T11:58:29.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.469+0000 7fa5f3aceb80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a71988c10 con 0x562a71a2fa90 2023-10-14T11:58:29.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.469+0000 7fa5f3aceb80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x562a71986530 con 0x562a71a29c90 2023-10-14T11:58:29.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.469+0000 7fa5e5680700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a71a2f6b0 0x562a71a6d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:29.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.469+0000 7fa5e5680700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a71a2f6b0 0x562a71a6d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45166/0 (socket says 172.21.15.78:45166) 2023-10-14T11:58:29.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.469+0000 7fa5e5680700 1 -- 172.21.15.78:0/3201284121 learned_addr learned my addr 172.21.15.78:0/3201284121 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:29.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.469+0000 7fa5e4e7f700 1 --2- 172.21.15.78:0/3201284121 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562a71a2fa90 0x562a71a6db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:29.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.469+0000 7fa5e5e81700 1 --2- 172.21.15.78:0/3201284121 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x562a71a29c90 0x562a71a4d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:29.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.469+0000 7fa5e5680700 1 -- 172.21.15.78:0/3201284121 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x562a71a29c90 msgr2=0x562a71a4d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:29.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.469+0000 7fa5e5680700 1 --2- 172.21.15.78:0/3201284121 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x562a71a29c90 0x562a71a4d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.469+0000 7fa5e5680700 1 -- 172.21.15.78:0/3201284121 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562a71a2fa90 msgr2=0x562a71a6db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:29.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.469+0000 7fa5e5680700 1 --2- 172.21.15.78:0/3201284121 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562a71a2fa90 0x562a71a6db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.469+0000 7fa5e5680700 1 -- 172.21.15.78:0/3201284121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562a7193f3a0 con 0x562a71a2f6b0 2023-10-14T11:58:29.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.469+0000 7fa5e5680700 1 --2- 172.21.15.78:0/3201284121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a71a2f6b0 0x562a71a6d620 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa5e00aada0 tx=0x7fa5d0010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=336eae9e015a0bba server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:29.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.470+0000 7fa5c6ffd700 1 -- 172.21.15.78:0/3201284121 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d0004200 con 0x562a71a2f6b0 2023-10-14T11:58:29.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.470+0000 7fa5c6ffd700 1 -- 172.21.15.78:0/3201284121 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d002b070 con 0x562a71a2f6b0 2023-10-14T11:58:29.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.470+0000 7fa5c6ffd700 1 -- 172.21.15.78:0/3201284121 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d0004560 con 0x562a71a2f6b0 2023-10-14T11:58:29.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.470+0000 7fa5f3aceb80 1 -- 172.21.15.78:0/3201284121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a71a2f6b0 msgr2=0x562a71a6d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:29.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.470+0000 7fa5f3aceb80 1 --2- 172.21.15.78:0/3201284121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a71a2f6b0 0x562a71a6d620 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa5e00aada0 tx=0x7fa5d0010000 comp rx=0 tx=0).stop 2023-10-14T11:58:29.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.470+0000 7fa5f3aceb80 1 -- 172.21.15.78:0/3201284121 shutdown_connections 2023-10-14T11:58:29.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.470+0000 7fa5f3aceb80 1 --2- 172.21.15.78:0/3201284121 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x562a71a29c90 0x562a71a4d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.470+0000 7fa5f3aceb80 1 --2- 172.21.15.78:0/3201284121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a71a2f6b0 0x562a71a6d620 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.470+0000 7fa5f3aceb80 1 --2- 172.21.15.78:0/3201284121 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562a71a2fa90 0x562a71a6db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.470+0000 7fa5f3aceb80 1 -- 172.21.15.78:0/3201284121 >> 172.21.15.78:0/3201284121 conn(0x562a7195cd80 msgr2=0x562a71a2d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:29.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.470+0000 7fa5f3aceb80 1 -- 172.21.15.78:0/3201284121 shutdown_connections 2023-10-14T11:58:29.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.470+0000 7fa5f3aceb80 1 -- 172.21.15.78:0/3201284121 wait complete. 2023-10-14T11:58:29.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.471+0000 7fa5f3aceb80 1 -- start start 2023-10-14T11:58:29.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.471+0000 7fa5f3aceb80 1 Processor -- start 2023-10-14T11:58:29.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.472+0000 7fa5f3aceb80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x562a71a29c90 0x562a71a7e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:29.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.472+0000 7fa5f3aceb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a71a2f6b0 0x562a71a7eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:29.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.472+0000 7fa5f3aceb80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562a71a2fa90 0x562a71a53fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:29.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.472+0000 7fa5f3aceb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a71981e20 con 0x562a71a2f6b0 2023-10-14T11:58:29.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.472+0000 7fa5f3aceb80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a7191e6c0 con 0x562a71a2fa90 2023-10-14T11:58:29.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.472+0000 7fa5f3aceb80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x562a7191e360 con 0x562a71a29c90 2023-10-14T11:58:29.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.472+0000 7fa5f3aceb80 10 client.0 _ll_register_callbacks cb 0x562a7177b7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T11:58:29.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.472+0000 7fa5e4e7f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a71a2f6b0 0x562a71a7eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:29.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.472+0000 7fa5e4e7f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a71a2f6b0 0x562a71a7eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45174/0 (socket says 172.21.15.78:45174) 2023-10-14T11:58:29.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.472+0000 7fa5e4e7f700 1 -- 172.21.15.78:0/243223377 learned_addr learned my addr 172.21.15.78:0/243223377 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:29.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.472+0000 7fa5e5e81700 1 --2- 172.21.15.78:0/243223377 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562a71a2fa90 0x562a71a53fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:29.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.472+0000 7fa5e5680700 1 --2- 172.21.15.78:0/243223377 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x562a71a29c90 0x562a71a7e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:29.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.472+0000 7fa5e4e7f700 1 -- 172.21.15.78:0/243223377 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x562a71a29c90 msgr2=0x562a71a7e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:29.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.472+0000 7fa5e4e7f700 1 --2- 172.21.15.78:0/243223377 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x562a71a29c90 0x562a71a7e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.472+0000 7fa5e4e7f700 1 -- 172.21.15.78:0/243223377 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562a71a2fa90 msgr2=0x562a71a53fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:29.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.472+0000 7fa5e4e7f700 1 --2- 172.21.15.78:0/243223377 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562a71a2fa90 0x562a71a53fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.472+0000 7fa5e4e7f700 1 -- 172.21.15.78:0/243223377 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5d0022050 con 0x562a71a2f6b0 2023-10-14T11:58:29.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.472+0000 7fa5e4e7f700 1 --2- 172.21.15.78:0/243223377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a71a2f6b0 0x562a71a7eb40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa5d40011e0 tx=0x7fa5d4001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=163c969e6e96fdc7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:29.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.472+0000 7fa5bffff700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-14T11:58:29.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.472+0000 7fa5bffff700 1 -- 172.21.15.78:0/243223377 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d4007760 con 0x562a71a2f6b0 2023-10-14T11:58:29.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.472+0000 7fa5bffff700 1 -- 172.21.15.78:0/243223377 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa5d400c780 con 0x562a71a2f6b0 2023-10-14T11:58:29.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.472+0000 7fa5f3aceb80 1 -- 172.21.15.78:0/243223377 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562a71a225b0 con 0x562a71a2f6b0 2023-10-14T11:58:29.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.473+0000 7fa5bffff700 1 -- 172.21.15.78:0/243223377 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5d400c940 con 0x562a71a2f6b0 2023-10-14T11:58:29.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.473+0000 7fa5f3aceb80 10 client.4632 fetch_fsmap learned FSMap version 8 2023-10-14T11:58:29.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.473+0000 7fa5f3aceb80 1 -- 172.21.15.78:0/243223377 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x562a717acd90 con 0x562a71a2f6b0 2023-10-14T11:58:29.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.473+0000 7fa5bffff700 1 -- 172.21.15.78:0/243223377 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d4024020 con 0x562a71a2f6b0 2023-10-14T11:58:29.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.473+0000 7fa5bffff700 1 -- 172.21.15.78:0/243223377 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 8) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fa5d400caa0 con 0x562a71a2f6b0 2023-10-14T11:58:29.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.473+0000 7fa5f3aceb80 10 client.4632 fetch_fsmap finished waiting for FSMap version 8 2023-10-14T11:58:29.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.473+0000 7fa5f3aceb80 10 client.4632 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-14T11:58:29.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.473+0000 7fa5f3aceb80 20 client.4632 populate_metadata read hostname 'smithi078' 2023-10-14T11:58:29.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.473+0000 7fa5f3aceb80 1 --2- 172.21.15.78:0/243223377 >> [v2:172.21.15.78:6835/920300787,v1:172.21.15.78:6837/920300787] conn(0x562a71a7acc0 0x562a71a7b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:29.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.473+0000 7fa5f3aceb80 4 client.4632 mds_command: new command op to 4269 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-14T11:58:29.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.473+0000 7fa5f3aceb80 1 -- 172.21.15.78:0/243223377 --> [v2:172.21.15.78:6835/920300787,v1:172.21.15.78:6837/920300787] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562a7197ae00 con 0x562a71a7acc0 2023-10-14T11:58:29.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.473+0000 7fa5e5680700 1 --2- 172.21.15.78:0/243223377 >> [v2:172.21.15.78:6835/920300787,v1:172.21.15.78:6837/920300787] conn(0x562a71a7acc0 0x562a71a7b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:29.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.473+0000 7fa5bffff700 10 client.4632 ms_handle_connect on v2:172.21.15.78:6835/920300787 2023-10-14T11:58:29.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.473+0000 7fa5e5680700 1 --2- 172.21.15.78:0/243223377 >> [v2:172.21.15.78:6835/920300787,v1:172.21.15.78:6837/920300787] conn(0x562a71a7acc0 0x562a71a7b0a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3628c98286e8c213 server_cookie=b4ddf44fdc3055d7 in_seq=0 out_seq=0 2023-10-14T11:58:29.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.474+0000 7fa5bffff700 1 -- 172.21.15.78:0/243223377 <== mds.0 v2:172.21.15.78:6835/920300787 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x562a7197ae00 con 0x562a71a7acc0 2023-10-14T11:58:29.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.474+0000 7fa5bffff700 10 client.4632 handle_command_reply: tid=0 2023-10-14T11:58:29.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.474+0000 7fa5f3aceb80 1 client.4632 shutdown 2023-10-14T11:58:29.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.474+0000 7fa5f3aceb80 1 -- 172.21.15.78:0/243223377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a71a2f6b0 msgr2=0x562a71a7eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:29.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.474+0000 7fa5f3aceb80 1 --2- 172.21.15.78:0/243223377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a71a2f6b0 0x562a71a7eb40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa5d40011e0 tx=0x7fa5d4001650 comp rx=0 tx=0).stop 2023-10-14T11:58:29.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.474+0000 7fa5f3aceb80 1 -- 172.21.15.78:0/243223377 shutdown_connections 2023-10-14T11:58:29.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.474+0000 7fa5f3aceb80 1 --2- 172.21.15.78:0/243223377 >> [v2:172.21.15.78:6835/920300787,v1:172.21.15.78:6837/920300787] conn(0x562a71a7acc0 0x562a71a7b0a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.474+0000 7fa5f3aceb80 1 --2- 172.21.15.78:0/243223377 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562a71a2fa90 0x562a71a53fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.474+0000 7fa5bffff700 0 client.4632 ms_handle_reset on v2:172.21.15.78:6835/920300787 2023-10-14T11:58:29.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.474+0000 7fa5f3aceb80 1 --2- 172.21.15.78:0/243223377 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x562a71a29c90 0x562a71a7e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.474+0000 7fa5f3aceb80 1 --2- 172.21.15.78:0/243223377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a71a2f6b0 0x562a71a7eb40 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.474+0000 7fa5f3aceb80 1 -- 172.21.15.78:0/243223377 >> 172.21.15.78:0/243223377 conn(0x562a7195cd80 msgr2=0x562a71a23f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:29.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.474+0000 7fa5f3aceb80 1 -- 172.21.15.78:0/243223377 shutdown_connections 2023-10-14T11:58:29.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.474+0000 7fa5f3aceb80 1 -- 172.21.15.78:0/243223377 wait complete. 2023-10-14T11:58:29.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.474+0000 7fa5f3aceb80 20 client.4632 trim_cache size 0 max 16384 2023-10-14T11:58:29.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.492+0000 7fa5f3aceb80 1 Processor -- start 2023-10-14T11:58:29.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.492+0000 7fa5f3aceb80 1 -- start start 2023-10-14T11:58:29.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.492+0000 7fa5f3aceb80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562a71a29c90 0x562a71a303b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:29.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.492+0000 7fa5f3aceb80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x562a71a2f6b0 0x562a71a4b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:29.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.492+0000 7fa5f3aceb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a71a2fa90 0x562a71a4bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:29.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.492+0000 7fa5f3aceb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a718028d0 con 0x562a71a2fa90 2023-10-14T11:58:29.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.492+0000 7fa5f3aceb80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a71a69360 con 0x562a71a29c90 2023-10-14T11:58:29.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.492+0000 7fa5f3aceb80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x562a71a05320 con 0x562a71a2f6b0 2023-10-14T11:58:29.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.492+0000 7fa5e5e81700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a71a2fa90 0x562a71a4bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:29.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.492+0000 7fa5e5e81700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a71a2fa90 0x562a71a4bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45184/0 (socket says 172.21.15.78:45184) 2023-10-14T11:58:29.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.492+0000 7fa5e5e81700 1 -- 172.21.15.78:0/3933740137 learned_addr learned my addr 172.21.15.78:0/3933740137 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:29.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.492+0000 7fa5e5680700 1 --2- 172.21.15.78:0/3933740137 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562a71a29c90 0x562a71a303b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:29.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.493+0000 7fa5e4e7f700 1 --2- 172.21.15.78:0/3933740137 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x562a71a2f6b0 0x562a71a4b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:29.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.493+0000 7fa5e5e81700 1 -- 172.21.15.78:0/3933740137 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x562a71a2f6b0 msgr2=0x562a71a4b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:29.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.493+0000 7fa5e5e81700 1 --2- 172.21.15.78:0/3933740137 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x562a71a2f6b0 0x562a71a4b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.493+0000 7fa5e5e81700 1 -- 172.21.15.78:0/3933740137 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562a71a29c90 msgr2=0x562a71a303b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:29.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.493+0000 7fa5e5e81700 1 --2- 172.21.15.78:0/3933740137 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562a71a29c90 0x562a71a303b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.493+0000 7fa5e5e81700 1 -- 172.21.15.78:0/3933740137 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562a71795c10 con 0x562a71a2fa90 2023-10-14T11:58:29.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.493+0000 7fa5e4e7f700 1 --2- 172.21.15.78:0/3933740137 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x562a71a2f6b0 0x562a71a4b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:29.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.493+0000 7fa5e5e81700 1 --2- 172.21.15.78:0/3933740137 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a71a2fa90 0x562a71a4bb50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa5d8057e00 tx=0x7fa5d805e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75faca4df00625cf server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:29.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.493+0000 7fa5c77fe700 1 -- 172.21.15.78:0/3933740137 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d805bfc0 con 0x562a71a2fa90 2023-10-14T11:58:29.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.493+0000 7fa5c77fe700 1 -- 172.21.15.78:0/3933740137 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d8060170 con 0x562a71a2fa90 2023-10-14T11:58:29.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.493+0000 7fa5c77fe700 1 -- 172.21.15.78:0/3933740137 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d8013e10 con 0x562a71a2fa90 2023-10-14T11:58:29.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.493+0000 7fa5f3aceb80 1 -- 172.21.15.78:0/3933740137 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a71a2fa90 msgr2=0x562a71a4bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:29.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.493+0000 7fa5f3aceb80 1 --2- 172.21.15.78:0/3933740137 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a71a2fa90 0x562a71a4bb50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa5d8057e00 tx=0x7fa5d805e040 comp rx=0 tx=0).stop 2023-10-14T11:58:29.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.493+0000 7fa5f3aceb80 1 -- 172.21.15.78:0/3933740137 shutdown_connections 2023-10-14T11:58:29.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.493+0000 7fa5f3aceb80 1 --2- 172.21.15.78:0/3933740137 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a71a2fa90 0x562a71a4bb50 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.493+0000 7fa5f3aceb80 1 --2- 172.21.15.78:0/3933740137 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562a71a29c90 0x562a71a303b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.493+0000 7fa5f3aceb80 1 --2- 172.21.15.78:0/3933740137 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x562a71a2f6b0 0x562a71a4b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.493+0000 7fa5f3aceb80 1 -- 172.21.15.78:0/3933740137 >> 172.21.15.78:0/3933740137 conn(0x562a7195cd80 msgr2=0x562a71a2dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:29.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.493+0000 7fa5f3aceb80 1 -- 172.21.15.78:0/3933740137 shutdown_connections 2023-10-14T11:58:29.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.493+0000 7fa5f3aceb80 1 -- 172.21.15.78:0/3933740137 wait complete. 2023-10-14T11:58:29.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.494+0000 7fa5f3aceb80 1 -- start start 2023-10-14T11:58:29.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.494+0000 7fa5f3aceb80 1 Processor -- start 2023-10-14T11:58:29.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.495+0000 7fa5f3aceb80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562a71a29c90 0x562a71acb230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:29.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.495+0000 7fa5f3aceb80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x562a71a2f6b0 0x562a71a8b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:29.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.495+0000 7fa5f3aceb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a71a2fa90 0x562a71ad76d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:29.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.495+0000 7fa5f3aceb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a71a514c0 con 0x562a71a2fa90 2023-10-14T11:58:29.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.495+0000 7fa5f3aceb80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a71ad3da0 con 0x562a71a29c90 2023-10-14T11:58:29.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.495+0000 7fa5f3aceb80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x562a71a50940 con 0x562a71a2f6b0 2023-10-14T11:58:29.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.495+0000 7fa5f3aceb80 10 client.0 _ll_register_callbacks cb 0x7fa5d4007060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T11:58:29.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.495+0000 7fa5e5e81700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a71a2fa90 0x562a71ad76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:29.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.495+0000 7fa5e5e81700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a71a2fa90 0x562a71ad76d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45190/0 (socket says 172.21.15.78:45190) 2023-10-14T11:58:29.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.495+0000 7fa5e5e81700 1 -- 172.21.15.78:0/677186024 learned_addr learned my addr 172.21.15.78:0/677186024 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:29.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.495+0000 7fa5e5680700 1 --2- 172.21.15.78:0/677186024 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562a71a29c90 0x562a71acb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:29.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.495+0000 7fa5e4e7f700 1 --2- 172.21.15.78:0/677186024 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x562a71a2f6b0 0x562a71a8b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:29.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.495+0000 7fa5e5e81700 1 -- 172.21.15.78:0/677186024 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x562a71a2f6b0 msgr2=0x562a71a8b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:29.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.495+0000 7fa5e5e81700 1 --2- 172.21.15.78:0/677186024 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x562a71a2f6b0 0x562a71a8b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.495+0000 7fa5e5e81700 1 -- 172.21.15.78:0/677186024 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562a71a29c90 msgr2=0x562a71acb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:29.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.495+0000 7fa5e5e81700 1 --2- 172.21.15.78:0/677186024 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562a71a29c90 0x562a71acb230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.495+0000 7fa5e5e81700 1 -- 172.21.15.78:0/677186024 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5e01e80c0 con 0x562a71a2fa90 2023-10-14T11:58:29.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.495+0000 7fa5e4e7f700 1 --2- 172.21.15.78:0/677186024 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x562a71a2f6b0 0x562a71a8b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:29.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.495+0000 7fa5e5e81700 1 --2- 172.21.15.78:0/677186024 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a71a2fa90 0x562a71ad76d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa5d800ef20 tx=0x7fa5d8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21eec98d802a5182 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:29.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.495+0000 7fa5bffff700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-14T11:58:29.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.495+0000 7fa5bffff700 1 -- 172.21.15.78:0/677186024 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d807a070 con 0x562a71a2fa90 2023-10-14T11:58:29.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.495+0000 7fa5bffff700 1 -- 172.21.15.78:0/677186024 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa5d805bfc0 con 0x562a71a2fa90 2023-10-14T11:58:29.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.495+0000 7fa5f3aceb80 1 -- 172.21.15.78:0/677186024 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa5d002b070 con 0x562a71a2fa90 2023-10-14T11:58:29.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.496+0000 7fa5bffff700 1 -- 172.21.15.78:0/677186024 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5d8013e10 con 0x562a71a2fa90 2023-10-14T11:58:29.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.496+0000 7fa5f3aceb80 10 client.4638 fetch_fsmap learned FSMap version 8 2023-10-14T11:58:29.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.496+0000 7fa5f3aceb80 1 -- 172.21.15.78:0/677186024 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x562a717d3200 con 0x562a71a2fa90 2023-10-14T11:58:29.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.496+0000 7fa5bffff700 1 -- 172.21.15.78:0/677186024 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d002b070 con 0x562a71a2fa90 2023-10-14T11:58:29.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.496+0000 7fa5bffff700 1 -- 172.21.15.78:0/677186024 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 8) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fa5d80811f0 con 0x562a71a2fa90 2023-10-14T11:58:29.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.496+0000 7fa5f3aceb80 10 client.4638 fetch_fsmap finished waiting for FSMap version 8 2023-10-14T11:58:29.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.496+0000 7fa5f3aceb80 10 client.4638 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-14T11:58:29.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.496+0000 7fa5f3aceb80 20 client.4638 populate_metadata read hostname 'smithi078' 2023-10-14T11:58:29.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.496+0000 7fa5f3aceb80 1 --2- 172.21.15.78:0/677186024 >> [v2:172.21.15.78:6835/920300787,v1:172.21.15.78:6837/920300787] conn(0x562a71a8ce70 0x562a71a8d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:29.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.496+0000 7fa5f3aceb80 4 client.4638 mds_command: new command op to 4269 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-14T11:58:29.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.496+0000 7fa5f3aceb80 1 -- 172.21.15.78:0/677186024 --> [v2:172.21.15.78:6835/920300787,v1:172.21.15.78:6837/920300787] -- command(tid 0: {"prefix": "status"}) v1 -- 0x562a7197ab20 con 0x562a71a8ce70 2023-10-14T11:58:29.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.496+0000 7fa5e5680700 1 --2- 172.21.15.78:0/677186024 >> [v2:172.21.15.78:6835/920300787,v1:172.21.15.78:6837/920300787] conn(0x562a71a8ce70 0x562a71a8d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:29.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.496+0000 7fa5bffff700 10 client.4638 ms_handle_connect on v2:172.21.15.78:6835/920300787 2023-10-14T11:58:29.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.496+0000 7fa5e5680700 1 --2- 172.21.15.78:0/677186024 >> [v2:172.21.15.78:6835/920300787,v1:172.21.15.78:6837/920300787] conn(0x562a71a8ce70 0x562a71a8d250 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32683c4ae8fc85af server_cookie=a1028bff5862a70e in_seq=0 out_seq=0 2023-10-14T11:58:29.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.496+0000 7fa5bffff700 1 -- 172.21.15.78:0/677186024 <== mds.0 v2:172.21.15.78:6835/920300787 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x562a7197ab20 con 0x562a71a8ce70 2023-10-14T11:58:29.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.496+0000 7fa5bffff700 10 client.4638 handle_command_reply: tid=0 2023-10-14T11:58:29.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.496+0000 7fa5f3aceb80 1 client.4638 shutdown 2023-10-14T11:58:29.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.497+0000 7fa5f3aceb80 1 -- 172.21.15.78:0/677186024 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a71a2fa90 msgr2=0x562a71ad76d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:29.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.497+0000 7fa5f3aceb80 1 --2- 172.21.15.78:0/677186024 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a71a2fa90 0x562a71ad76d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa5d800ef20 tx=0x7fa5d8009f90 comp rx=0 tx=0).stop 2023-10-14T11:58:29.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.497+0000 7fa5f3aceb80 1 -- 172.21.15.78:0/677186024 shutdown_connections 2023-10-14T11:58:29.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.497+0000 7fa5f3aceb80 1 --2- 172.21.15.78:0/677186024 >> [v2:172.21.15.78:6835/920300787,v1:172.21.15.78:6837/920300787] conn(0x562a71a8ce70 0x562a71a8d250 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.497+0000 7fa5f3aceb80 1 --2- 172.21.15.78:0/677186024 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a71a2fa90 0x562a71ad76d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.497+0000 7fa5bffff700 0 client.4638 ms_handle_reset on v2:172.21.15.78:6835/920300787 2023-10-14T11:58:29.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.497+0000 7fa5f3aceb80 1 --2- 172.21.15.78:0/677186024 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562a71a29c90 0x562a71acb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.497+0000 7fa5f3aceb80 1 --2- 172.21.15.78:0/677186024 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x562a71a2f6b0 0x562a71a8b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.497+0000 7fa5f3aceb80 1 -- 172.21.15.78:0/677186024 >> 172.21.15.78:0/677186024 conn(0x562a7195cd80 msgr2=0x562a71a6c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:29.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.497+0000 7fa5f3aceb80 1 -- 172.21.15.78:0/677186024 shutdown_connections 2023-10-14T11:58:29.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.497+0000 7fa5f3aceb80 1 -- 172.21.15.78:0/677186024 wait complete. 2023-10-14T11:58:29.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.497+0000 7fa5f3aceb80 20 client.4638 trim_cache size 0 max 16384 2023-10-14T11:58:29.528 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-14T11:58:29.528 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "4c5dc134-3092-406a-a1e1-13e6d9e926e8", 2023-10-14T11:58:29.528 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2023-10-14T11:58:29.528 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 4269, 2023-10-14T11:58:29.529 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2023-10-14T11:58:29.529 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2023-10-14T11:58:29.529 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2023-10-14T11:58:29.529 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 14.349085283000001, 2023-10-14T11:58:29.529 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 8, 2023-10-14T11:58:29.530 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 21, 2023-10-14T11:58:29.530 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 21, 2023-10-14T11:58:29.530 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 21.143341973999998 2023-10-14T11:58:29.530 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-14T11:58:29.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.497+0000 7fa5e78e4700 1 -- 172.21.15.78:0/2258725310 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa5cc06af00 msgr2=0x7fa5cc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:29.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.497+0000 7fa5e78e4700 1 --2- 172.21.15.78:0/2258725310 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa5cc06af00 0x7fa5cc06d3c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0014040 tx=0x7fa5d0011000 comp rx=0 tx=0).stop 2023-10-14T11:58:29.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.497+0000 7fa5e78e4700 1 -- 172.21.15.78:0/2258725310 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5e01500a0 msgr2=0x7fa5e01eb560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:29.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.497+0000 7fa5e78e4700 1 --2- 172.21.15.78:0/2258725310 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5e01500a0 0x7fa5e01eb560 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa5d800e810 tx=0x7fa5d800eb20 comp rx=0 tx=0).stop 2023-10-14T11:58:29.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.497+0000 7fa5e78e4700 1 -- 172.21.15.78:0/2258725310 shutdown_connections 2023-10-14T11:58:29.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.497+0000 7fa5e78e4700 1 --2- 172.21.15.78:0/2258725310 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5e01500a0 0x7fa5e01eb560 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.497+0000 7fa5e78e4700 1 --2- 172.21.15.78:0/2258725310 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa5e00ad550 0x7fa5e00c18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.497+0000 7fa5e78e4700 1 --2- 172.21.15.78:0/2258725310 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa5cc06af00 0x7fa5cc06d3c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.497+0000 7fa5e78e4700 1 --2- 172.21.15.78:0/2258725310 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5e00adaa0 0x7fa5e01eb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.497+0000 7fa5e78e4700 1 -- 172.21.15.78:0/2258725310 >> 172.21.15.78:0/2258725310 conn(0x7fa5e0072a00 msgr2=0x7fa5e0151c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:29.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.497+0000 7fa5e78e4700 1 -- 172.21.15.78:0/2258725310 shutdown_connections 2023-10-14T11:58:29.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.497+0000 7fa5e78e4700 1 -- 172.21.15.78:0/2258725310 wait complete. 2023-10-14T11:58:29.535 INFO:teuthology.run_tasks:Running task kclient... 2023-10-14T11:58:29.558 INFO:tasks.kclient:Mounting kernel clients... 2023-10-14T11:58:29.559 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2023-10-14T11:58:29.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:29.560 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2023-10-14T11:58:29.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:29.577 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link delete ceph-brx 2023-10-14T11:58:29.645 INFO:teuthology.orchestra.run.smithi078.stderr:Cannot find device "ceph-brx" 2023-10-14T11:58:29.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T11:58:29.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:29.647 DEBUG:teuthology.orchestra.run.smithi195:> ip netns list 2023-10-14T11:58:29.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:29.669 DEBUG:teuthology.orchestra.run.smithi195:> sudo ip link delete ceph-brx 2023-10-14T11:58:29.743 INFO:teuthology.orchestra.run.smithi195.stderr:Cannot find device "ceph-brx" 2023-10-14T11:58:29.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T11:58:29.745 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2023-10-14T11:58:29.745 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-14T11:58:29.745 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-14T11:58:29.746 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-14T11:58:29.866 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.865+0000 7f1089c4e700 1 Processor -- start 2023-10-14T11:58:29.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.865+0000 7f1089c4e700 1 -- start start 2023-10-14T11:58:29.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.865+0000 7f1089c4e700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f108406ab40 0x7f108406f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:29.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.865+0000 7f1089c4e700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f108406b630 con 0x7f10840c0350 2023-10-14T11:58:29.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.865+0000 7f1089c4e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10840c49f0 con 0x7f10841554e0 2023-10-14T11:58:29.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.865+0000 7f1089c4e700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10840726a0 con 0x7f108406ab40 2023-10-14T11:58:29.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.866+0000 7f1083fff700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f10840c0350 0x7f108413d7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50074/0 (socket says 172.21.15.78:50074) 2023-10-14T11:58:29.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.866+0000 7f1083fff700 1 -- 172.21.15.78:0/2388342950 learned_addr learned my addr 172.21.15.78:0/2388342950 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:29.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.866+0000 7f10837fe700 1 --2- 172.21.15.78:0/2388342950 >> v2:172.21.15.195:3300/0 conn(0x7f108406ab40 0x7f108406f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:29.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.866+0000 7f10827fc700 1 -- 172.21.15.78:0/2388342950 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2799651207 0 0) 0x7f108406b630 con 0x7f10840c0350 2023-10-14T11:58:29.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.866+0000 7f10827fc700 1 -- 172.21.15.78:0/2388342950 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f106c0035b0 con 0x7f10840c0350 2023-10-14T11:58:29.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.866+0000 7f10827fc700 1 -- 172.21.15.78:0/2388342950 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 620460610 0 0) 0x7f106c0035b0 con 0x7f10840c0350 2023-10-14T11:58:29.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.866+0000 7f10827fc700 1 -- 172.21.15.78:0/2388342950 >> v2:172.21.15.195:3300/0 conn(0x7f108406ab40 msgr2=0x7f108406f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:29.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.866+0000 7f10827fc700 1 --2- 172.21.15.78:0/2388342950 >> v2:172.21.15.195:3300/0 conn(0x7f108406ab40 0x7f108406f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.873 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.866+0000 7f10827fc700 1 -- 172.21.15.78:0/2388342950 >> v1:172.21.15.195:6789/0 conn(0x7f10841554e0 legacy=0x7f108406af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:29.873 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.866+0000 7f10827fc700 1 -- 172.21.15.78:0/2388342950 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1084072980 con 0x7f10840c0350 2023-10-14T11:58:29.873 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.866+0000 7f10827fc700 1 -- 172.21.15.78:0/2388342950 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441799917 0 0) 0x7f107c0019f0 con 0x7f10840c0350 2023-10-14T11:58:29.873 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.866+0000 7f10827fc700 1 -- 172.21.15.78:0/2388342950 >> v1:172.21.15.78:6789/0 conn(0x7f10840c0350 legacy=0x7f108413d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:29.873 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.866+0000 7f10827fc700 1 --2- 172.21.15.78:0/2388342950 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f106c003a00 0x7f106c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:29.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.866+0000 7f10827fc700 1 --2- 172.21.15.78:0/2388342950 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f106c004850 0x7f106c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:29.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.866+0000 7f10827fc700 1 --2- 172.21.15.78:0/2388342950 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f106c005210 0x7f106c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:29.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.866+0000 7f10827fc700 1 -- 172.21.15.78:0/2388342950 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10840726a0 con 0x7f106c004850 2023-10-14T11:58:29.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.866+0000 7f10827fc700 1 -- 172.21.15.78:0/2388342950 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106c00c1b0 con 0x7f106c005210 2023-10-14T11:58:29.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.866+0000 7f10827fc700 1 -- 172.21.15.78:0/2388342950 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f106c00c370 con 0x7f106c003a00 2023-10-14T11:58:29.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.866+0000 7f1082ffd700 1 --2- 172.21.15.78:0/2388342950 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f106c004850 0x7f106c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:29.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.866+0000 7f1083fff700 1 --2- 172.21.15.78:0/2388342950 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f106c005210 0x7f106c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:29.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.866+0000 7f10837fe700 1 --2- 172.21.15.78:0/2388342950 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f106c003a00 0x7f106c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:29.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.866+0000 7f1082ffd700 1 -- 172.21.15.78:0/2388342950 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f106c003a00 msgr2=0x7f106c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:29.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.866+0000 7f1082ffd700 1 --2- 172.21.15.78:0/2388342950 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f106c003a00 0x7f106c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.866+0000 7f1082ffd700 1 -- 172.21.15.78:0/2388342950 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f106c005210 msgr2=0x7f106c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:29.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.866+0000 7f1082ffd700 1 --2- 172.21.15.78:0/2388342950 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f106c005210 0x7f106c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.866+0000 7f1082ffd700 1 -- 172.21.15.78:0/2388342950 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106c0035b0 con 0x7f106c004850 2023-10-14T11:58:29.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.866+0000 7f1083fff700 1 --2- 172.21.15.78:0/2388342950 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f106c005210 0x7f106c00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:29.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.866+0000 7f1083fff700 1 -- 172.21.15.78:0/2388342950 reap_dead start 2023-10-14T11:58:29.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.867+0000 7f1082ffd700 1 --2- 172.21.15.78:0/2388342950 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f106c004850 0x7f106c004cd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f10780017c0 tx=0x7f106c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8534fc1c1e558846 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:29.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.867+0000 7f10827fc700 1 -- 172.21.15.78:0/2388342950 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10780052d0 con 0x7f106c004850 2023-10-14T11:58:29.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.867+0000 7f10827fc700 1 -- 172.21.15.78:0/2388342950 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1078005430 con 0x7f106c004850 2023-10-14T11:58:29.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.867+0000 7f10827fc700 1 -- 172.21.15.78:0/2388342950 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10780038c0 con 0x7f106c004850 2023-10-14T11:58:29.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.867+0000 7f1089c4e700 1 -- 172.21.15.78:0/2388342950 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f106c004850 msgr2=0x7f106c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:29.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.867+0000 7f1089c4e700 1 --2- 172.21.15.78:0/2388342950 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f106c004850 0x7f106c004cd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f10780017c0 tx=0x7f106c005880 comp rx=0 tx=0).stop 2023-10-14T11:58:29.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.867+0000 7f1089c4e700 1 -- 172.21.15.78:0/2388342950 shutdown_connections 2023-10-14T11:58:29.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.867+0000 7f1089c4e700 1 --2- 172.21.15.78:0/2388342950 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f106c004850 0x7f106c004cd0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.867+0000 7f1089c4e700 1 -- 172.21.15.78:0/2388342950 >> 172.21.15.78:0/2388342950 conn(0x7f10840c3ac0 msgr2=0x7f10840c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:29.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.867+0000 7f1089c4e700 1 -- 172.21.15.78:0/2388342950 shutdown_connections 2023-10-14T11:58:29.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.867+0000 7f1089c4e700 1 -- 172.21.15.78:0/2388342950 wait complete. 2023-10-14T11:58:29.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.869+0000 7f1089c4e700 1 Processor -- start 2023-10-14T11:58:29.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.869+0000 7f1089c4e700 1 -- start start 2023-10-14T11:58:29.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.869+0000 7f1089c4e700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f106c004850 0x7f1084151990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:29.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.869+0000 7f1089c4e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f108413ab70 0x7f10841381f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:29.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.869+0000 7f1089c4e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1084157650 0x7f1084138730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:29.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.869+0000 7f1089c4e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10841503e0 con 0x7f108413ab70 2023-10-14T11:58:29.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.869+0000 7f1089c4e700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10841506c0 con 0x7f1084157650 2023-10-14T11:58:29.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.870+0000 7f1089c4e700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10841509a0 con 0x7f106c004850 2023-10-14T11:58:29.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.870+0000 7f1082ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f108413ab70 0x7f10841381f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:29.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.870+0000 7f1082ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f108413ab70 0x7f10841381f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45208/0 (socket says 172.21.15.78:45208) 2023-10-14T11:58:29.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.870+0000 7f1082ffd700 1 -- 172.21.15.78:0/2722423394 learned_addr learned my addr 172.21.15.78:0/2722423394 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:29.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.870+0000 7f10837fe700 1 --2- 172.21.15.78:0/2722423394 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f106c004850 0x7f1084151990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:29.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.870+0000 7f1083fff700 1 --2- 172.21.15.78:0/2722423394 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1084157650 0x7f1084138730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:29.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.870+0000 7f1082ffd700 1 -- 172.21.15.78:0/2722423394 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f106c004850 msgr2=0x7f1084151990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:29.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.870+0000 7f1082ffd700 1 --2- 172.21.15.78:0/2722423394 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f106c004850 0x7f1084151990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.870+0000 7f1082ffd700 1 -- 172.21.15.78:0/2722423394 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1084157650 msgr2=0x7f1084138730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:29.885 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.870+0000 7f1082ffd700 1 --2- 172.21.15.78:0/2722423394 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1084157650 0x7f1084138730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:29.885 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.870+0000 7f1082ffd700 1 -- 172.21.15.78:0/2722423394 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1084150c80 con 0x7f108413ab70 2023-10-14T11:58:29.885 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.870+0000 7f1082ffd700 1 --2- 172.21.15.78:0/2722423394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f108413ab70 0x7f10841381f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1078015040 tx=0x7f1078001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:29.885 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.870+0000 7f106bfff700 1 -- 172.21.15.78:0/2722423394 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1078001290 con 0x7f108413ab70 2023-10-14T11:58:29.885 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.870+0000 7f106bfff700 1 -- 172.21.15.78:0/2722423394 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f107801d480 con 0x7f108413ab70 2023-10-14T11:58:29.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.870+0000 7f1089c4e700 1 -- 172.21.15.78:0/2722423394 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1084150f60 con 0x7f108413ab70 2023-10-14T11:58:29.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.870+0000 7f106bfff700 1 -- 172.21.15.78:0/2722423394 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f107801d830 con 0x7f108413ab70 2023-10-14T11:58:29.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.870+0000 7f1089c4e700 1 -- 172.21.15.78:0/2722423394 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1084150000 con 0x7f108413ab70 2023-10-14T11:58:29.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.871+0000 7f1089c4e700 1 -- 172.21.15.78:0/2722423394 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1084155d30 con 0x7f108413ab70 2023-10-14T11:58:29.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.872+0000 7f106bfff700 1 -- 172.21.15.78:0/2722423394 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1078019040 con 0x7f108413ab70 2023-10-14T11:58:29.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.872+0000 7f106bfff700 1 --2- 172.21.15.78:0/2722423394 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f107006af30 0x7f107006d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:29.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.872+0000 7f10837fe700 1 --2- 172.21.15.78:0/2722423394 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f107006af30 0x7f107006d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:29.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.872+0000 7f106bfff700 1 -- 172.21.15.78:0/2722423394 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f107808a000 con 0x7f108413ab70 2023-10-14T11:58:29.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.872+0000 7f10837fe700 1 --2- 172.21.15.78:0/2722423394 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f107006af30 0x7f107006d3f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1074009be0 tx=0x7f1074008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:29.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:29.876+0000 7f106bfff700 1 -- 172.21.15.78:0/2722423394 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1084155d30 con 0x7f108413ab70 2023-10-14T11:58:30.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:30.110+0000 7f1089c4e700 1 -- 172.21.15.78:0/2722423394 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1084155d30 con 0x7f108413ab70 2023-10-14T11:58:30.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:30.111+0000 7f106bfff700 1 -- 172.21.15.78:0/2722423394 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f1078017060 con 0x7f108413ab70 2023-10-14T11:58:30.113 INFO:teuthology.orchestra.run.smithi078.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-14T11:58:30.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:30.113+0000 7f1089c4e700 1 -- 172.21.15.78:0/2722423394 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f107006af30 msgr2=0x7f107006d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:30.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:30.113+0000 7f1089c4e700 1 --2- 172.21.15.78:0/2722423394 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f107006af30 0x7f107006d3f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1074009be0 tx=0x7f1074008040 comp rx=0 tx=0).stop 2023-10-14T11:58:30.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:30.113+0000 7f1089c4e700 1 -- 172.21.15.78:0/2722423394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f108413ab70 msgr2=0x7f10841381f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:30.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:30.113+0000 7f1089c4e700 1 --2- 172.21.15.78:0/2722423394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f108413ab70 0x7f10841381f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1078015040 tx=0x7f1078001120 comp rx=0 tx=0).stop 2023-10-14T11:58:30.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:30.113+0000 7f1089c4e700 1 -- 172.21.15.78:0/2722423394 shutdown_connections 2023-10-14T11:58:30.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:30.113+0000 7f1089c4e700 1 --2- 172.21.15.78:0/2722423394 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f107006af30 0x7f107006d3f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:30.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:30.113+0000 7f1089c4e700 1 --2- 172.21.15.78:0/2722423394 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1084157650 0x7f1084138730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:30.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:30.113+0000 7f1089c4e700 1 --2- 172.21.15.78:0/2722423394 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f106c004850 0x7f1084151990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:30.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:30.113+0000 7f1089c4e700 1 --2- 172.21.15.78:0/2722423394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f108413ab70 0x7f10841381f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:30.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:30.113+0000 7f1089c4e700 1 -- 172.21.15.78:0/2722423394 >> 172.21.15.78:0/2722423394 conn(0x7f10840c3ac0 msgr2=0x7f108413baa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:30.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:30.113+0000 7f1089c4e700 1 -- 172.21.15.78:0/2722423394 shutdown_connections 2023-10-14T11:58:30.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:30.113+0000 7f1089c4e700 1 -- 172.21.15.78:0/2722423394 wait complete. 2023-10-14T11:58:30.127 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-14T11:58:30.127 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-14T11:58:30.127 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi078.front.sepia.ceph.com 2023-10-14T11:58:30.128 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-14T11:58:30.128 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-14T11:58:30.128 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-14T11:58:30.128 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-14T11:58:30.129 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-14T11:58:30.129 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-14T11:58:30.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:30.130 DEBUG:teuthology.orchestra.run.smithi078:> ip addr 2023-10-14T11:58:30.147 INFO:teuthology.orchestra.run.smithi078.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-14T11:58:30.147 INFO:teuthology.orchestra.run.smithi078.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-14T11:58:30.148 INFO:teuthology.orchestra.run.smithi078.stdout: inet 127.0.0.1/8 scope host lo 2023-10-14T11:58:30.148 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2023-10-14T11:58:30.148 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 ::1/128 scope host 2023-10-14T11:58:30.148 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2023-10-14T11:58:30.149 INFO:teuthology.orchestra.run.smithi078.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T11:58:30.149 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:22 brd ff:ff:ff:ff:ff:ff 2023-10-14T11:58:30.149 INFO:teuthology.orchestra.run.smithi078.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T11:58:30.149 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:23 brd ff:ff:ff:ff:ff:ff 2023-10-14T11:58:30.149 INFO:teuthology.orchestra.run.smithi078.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T11:58:30.150 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dc brd ff:ff:ff:ff:ff:ff 2023-10-14T11:58:30.150 INFO:teuthology.orchestra.run.smithi078.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-14T11:58:30.150 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dd brd ff:ff:ff:ff:ff:ff 2023-10-14T11:58:30.150 INFO:teuthology.orchestra.run.smithi078.stdout: inet 172.21.15.78/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-14T11:58:30.151 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft 42414sec preferred_lft 42414sec 2023-10-14T11:58:30.151 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 fe80::e6db:28c0:bb52:823d/64 scope link noprefixroute 2023-10-14T11:58:30.151 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2023-10-14T11:58:30.151 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-14T11:58:30.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:30.152 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2023-10-14T11:58:30.153 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2023-10-14T11:58:30.153 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link add name ceph-brx type bridge 2023-10-14T11:58:30.153 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip addr flush dev ceph-brx 2023-10-14T11:58:30.153 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set ceph-brx up 2023-10-14T11:58:30.154 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-14T11:58:30.154 DEBUG:teuthology.orchestra.run.smithi078:> ') 2023-10-14T11:58:30.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:30.315 DEBUG:teuthology.orchestra.run.smithi078:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-14T11:58:30.344 INFO:teuthology.orchestra.run.smithi078.stdout:1 2023-10-14T11:58:30.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:30.345 DEBUG:teuthology.orchestra.run.smithi078:> route 2023-10-14T11:58:30.435 INFO:teuthology.orchestra.run.smithi078.stdout:Kernel IP routing table 2023-10-14T11:58:30.436 INFO:teuthology.orchestra.run.smithi078.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-14T11:58:30.436 INFO:teuthology.orchestra.run.smithi078.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-14T11:58:30.436 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-14T11:58:30.436 INFO:teuthology.orchestra.run.smithi078.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-14T11:58:30.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:30.437 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2023-10-14T11:58:30.437 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2023-10-14T11:58:30.438 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-14T11:58:30.438 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-14T11:58:30.438 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-14T11:58:30.439 DEBUG:teuthology.orchestra.run.smithi078:> ') 2023-10-14T11:58:30.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:30.646 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2023-10-14T11:58:30.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:30.664 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list-id 2023-10-14T11:58:30.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:30.720 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2023-10-14T11:58:30.721 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2023-10-14T11:58:30.721 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-14T11:58:30.721 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-10-14T11:58:30.722 DEBUG:teuthology.orchestra.run.smithi078:> ') 2023-10-14T11:58:30.814 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-10-14T11:58:30.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:30.815 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2023-10-14T11:58:30.815 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2023-10-14T11:58:30.815 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-10-14T11:58:30.816 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-14T11:58:30.816 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-10-14T11:58:30.816 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-10-14T11:58:30.817 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-10-14T11:58:30.817 DEBUG:teuthology.orchestra.run.smithi078:> ') 2023-10-14T11:58:31.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:31.011 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2023-10-14T11:58:31.011 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2023-10-14T11:58:31.011 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set brx.0 up 2023-10-14T11:58:31.012 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set dev brx.0 master ceph-brx 2023-10-14T11:58:31.012 DEBUG:teuthology.orchestra.run.smithi078:> ') 2023-10-14T11:58:31.116 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T11:58:31.116 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-14T11:58:31.135 INFO:teuthology.orchestra.run.smithi078.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-14T11:58:31.135 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T11:58:31.135 DEBUG:teuthology.orchestra.run.smithi078:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-14T11:58:31.192 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-14T11:58:31.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:31.192 DEBUG:teuthology.orchestra.run.smithi078:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-14T11:58:31.304 INFO:teuthology.orchestra.run.smithi078.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-14T11:58:31.304 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-14T11:58:31.713 INFO:teuthology.orchestra.run.smithi078.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-14T11:58:31.713 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-14T11:58:31.713 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: resolved to: "172.21.15.78:3300,172.21.15.195:3300,172.21.15.195:3301" 2023-10-14T11:58:31.713 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: trying mount with new device syntax: 0@4c5dc134-3092-406a-a1e1-13e6d9e926e8.cephfs=/ 2023-10-14T11:58:31.714 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.78:3300/172.21.15.195:3300/172.21.15.195:3301" will pass to kernel 2023-10-14T11:58:31.714 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: v2 mount support check returned 2 2023-10-14T11:58:31.714 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-14T11:58:31.715 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T11:58:31.751 INFO:teuthology.orchestra.run.smithi078.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-14T11:58:31.751 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-14T11:58:31.752 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: resolved to: "172.21.15.78:3300,172.21.15.195:3300,172.21.15.195:3301" 2023-10-14T11:58:31.752 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: trying mount with new device syntax: 0@4c5dc134-3092-406a-a1e1-13e6d9e926e8.cephfs=/ 2023-10-14T11:58:31.752 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.78:3300/172.21.15.195:3300/172.21.15.195:3301" will pass to kernel 2023-10-14T11:58:31.752 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: v2 mount support check returned 2 2023-10-14T11:58:31.753 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-14T11:58:31.753 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T11:58:31.753 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: resolved to: "172.21.15.78:6789,172.21.15.195:6789,172.21.15.195:6790" 2023-10-14T11:58:31.753 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.78:6789,172.21.15.195:6789,172.21.15.195:6790:/ 2023-10-14T11:58:31.754 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,key=0,mds_namespace=cephfs,fsid=4c5dc134-3092-406a-a1e1-13e6d9e926e8" will pass to kernel 2023-10-14T11:58:31.768 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-14T11:58:31.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:31.769 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-14T11:58:31.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:31.797 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-14T11:58:31.879 INFO:teuthology.orchestra.run.smithi078.stdout:client4647 2023-10-14T11:58:31.879 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-14T11:58:31.880 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T11:58:31.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:31.939 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-14T11:58:32.019 INFO:teuthology.orchestra.run.smithi078.stdout:client4647 2023-10-14T11:58:32.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:32.020 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-14T11:58:32.101 INFO:teuthology.orchestra.run.smithi078.stdout:client4647 2023-10-14T11:58:32.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:32.101 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/4c5dc134-3092-406a-a1e1-13e6d9e926e8.client4647/status') 2023-10-14T11:58:32.189 INFO:teuthology.orchestra.run.smithi078.stdout:instance: client.4647 (3)192.168.0.1:0/3537576722 2023-10-14T11:58:32.190 INFO:teuthology.orchestra.run.smithi078.stdout:blocklisted: false 2023-10-14T11:58:32.190 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records in 2023-10-14T11:58:32.191 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records out 2023-10-14T11:58:32.191 INFO:teuthology.orchestra.run.smithi078.stderr:69 bytes copied, 2.0688e-05 s, 3.3 MB/s 2023-10-14T11:58:32.192 INFO:tasks.kclient:client.1 config is {'syntax': 'v2'} 2023-10-14T11:58:32.192 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-14T11:58:32.192 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-10-14T11:58:32.193 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-14T11:58:32.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.311+0000 7f608404e700 1 Processor -- start 2023-10-14T11:58:32.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.311+0000 7f608404e700 1 -- start start 2023-10-14T11:58:32.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.312+0000 7f608404e700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f607c06da80 0x7f607c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:32.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.312+0000 7f608404e700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f607c06de60 0x7f607c155310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:32.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.312+0000 7f608404e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f607c06e2a0 con 0x7f607c0ac1d0 2023-10-14T11:58:32.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.312+0000 7f608404e700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f607c0710d0 con 0x7f607c06da80 2023-10-14T11:58:32.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.312+0000 7f608404e700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f607c158900 con 0x7f607c06de60 2023-10-14T11:58:32.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.312+0000 7f6081dea700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f607c06da80 0x7f607c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:32.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.312+0000 7f6081dea700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f607c06da80 0x7f607c155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45228/0 (socket says 172.21.15.78:45228) 2023-10-14T11:58:32.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.312+0000 7f6081dea700 1 -- 172.21.15.78:0/1016479576 learned_addr learned my addr 172.21.15.78:0/1016479576 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:32.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.312+0000 7f60825eb700 1 --2- 172.21.15.78:0/1016479576 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f607c06de60 0x7f607c155310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:32.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.312+0000 7f6081dea700 1 -- 172.21.15.78:0/1016479576 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f607c06de60 msgr2=0x7f607c155310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:32.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.312+0000 7f6081dea700 1 --2- 172.21.15.78:0/1016479576 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f607c06de60 0x7f607c155310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:32.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.312+0000 7f6081dea700 1 -- 172.21.15.78:0/1016479576 >> v1:172.21.15.195:6789/0 conn(0x7f607c0ac1d0 legacy=0x7f607c1580b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:32.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.312+0000 7f6081dea700 1 -- 172.21.15.78:0/1016479576 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f607c158be0 con 0x7f607c06da80 2023-10-14T11:58:32.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.312+0000 7f6081dea700 1 --2- 172.21.15.78:0/1016479576 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f607c06da80 0x7f607c155fb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f60780098f0 tx=0x7f6078009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ced90500d855882 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:32.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.312+0000 7f6080de8700 1 -- 172.21.15.78:0/1016479576 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f607800b8a0 con 0x7f607c06da80 2023-10-14T11:58:32.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.312+0000 7f6080de8700 1 -- 172.21.15.78:0/1016479576 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f607800ba00 con 0x7f607c06da80 2023-10-14T11:58:32.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.312+0000 7f6080de8700 1 -- 172.21.15.78:0/1016479576 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f607800bd90 con 0x7f607c06da80 2023-10-14T11:58:32.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.313+0000 7f608404e700 1 -- 172.21.15.78:0/1016479576 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f607c06da80 msgr2=0x7f607c155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:32.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.313+0000 7f608404e700 1 --2- 172.21.15.78:0/1016479576 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f607c06da80 0x7f607c155fb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f60780098f0 tx=0x7f6078009c00 comp rx=0 tx=0).stop 2023-10-14T11:58:32.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.313+0000 7f608404e700 1 -- 172.21.15.78:0/1016479576 shutdown_connections 2023-10-14T11:58:32.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.313+0000 7f608404e700 1 --2- 172.21.15.78:0/1016479576 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f607c06de60 0x7f607c155310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:32.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.313+0000 7f608404e700 1 --2- 172.21.15.78:0/1016479576 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f607c06da80 0x7f607c155fb0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:32.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.313+0000 7f608404e700 1 -- 172.21.15.78:0/1016479576 >> 172.21.15.78:0/1016479576 conn(0x7f607c0a8410 msgr2=0x7f607c0a8820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:32.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.313+0000 7f608404e700 1 -- 172.21.15.78:0/1016479576 shutdown_connections 2023-10-14T11:58:32.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.313+0000 7f608404e700 1 -- 172.21.15.78:0/1016479576 wait complete. 2023-10-14T11:58:32.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.314+0000 7f608404e700 1 Processor -- start 2023-10-14T11:58:32.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.314+0000 7f608404e700 1 -- start start 2023-10-14T11:58:32.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.315+0000 7f608404e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f607c06da80 0x7f607c1530d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:32.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.315+0000 7f608404e700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f607c06de60 0x7f607c145280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:32.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.315+0000 7f608404e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f607c0ac1d0 0x7f607c1457c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:32.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.315+0000 7f608404e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607c145d00 con 0x7f607c06da80 2023-10-14T11:58:32.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.315+0000 7f608404e700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607c157d30 con 0x7f607c0ac1d0 2023-10-14T11:58:32.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.315+0000 7f608404e700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f607c1e9f70 con 0x7f607c06de60 2023-10-14T11:58:32.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.315+0000 7f6081dea700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f607c06da80 0x7f607c1530d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:32.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.315+0000 7f6081dea700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f607c06da80 0x7f607c1530d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45234/0 (socket says 172.21.15.78:45234) 2023-10-14T11:58:32.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.315+0000 7f6081dea700 1 -- 172.21.15.78:0/2708695857 learned_addr learned my addr 172.21.15.78:0/2708695857 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:32.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.315+0000 7f60825eb700 1 --2- 172.21.15.78:0/2708695857 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f607c0ac1d0 0x7f607c1457c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:32.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.315+0000 7f60815e9700 1 --2- 172.21.15.78:0/2708695857 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f607c06de60 0x7f607c145280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:32.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.315+0000 7f6081dea700 1 -- 172.21.15.78:0/2708695857 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f607c06de60 msgr2=0x7f607c145280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:32.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.315+0000 7f6081dea700 1 --2- 172.21.15.78:0/2708695857 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f607c06de60 0x7f607c145280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:32.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.315+0000 7f6081dea700 1 -- 172.21.15.78:0/2708695857 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f607c0ac1d0 msgr2=0x7f607c1457c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:32.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.315+0000 7f6081dea700 1 --2- 172.21.15.78:0/2708695857 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f607c0ac1d0 0x7f607c1457c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:32.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.316+0000 7f6081dea700 1 -- 172.21.15.78:0/2708695857 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6078009550 con 0x7f607c06da80 2023-10-14T11:58:32.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.316+0000 7f6081dea700 1 --2- 172.21.15.78:0/2708695857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f607c06da80 0x7f607c1530d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6078000c00 tx=0x7f6078004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:32.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.316+0000 7f60727fc700 1 -- 172.21.15.78:0/2708695857 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6078004300 con 0x7f607c06da80 2023-10-14T11:58:32.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.316+0000 7f60727fc700 1 -- 172.21.15.78:0/2708695857 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f607801f070 con 0x7f607c06da80 2023-10-14T11:58:32.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.316+0000 7f608404e700 1 -- 172.21.15.78:0/2708695857 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f607c1ea250 con 0x7f607c06da80 2023-10-14T11:58:32.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.316+0000 7f60727fc700 1 -- 172.21.15.78:0/2708695857 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60780036a0 con 0x7f607c06da80 2023-10-14T11:58:32.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.316+0000 7f608404e700 1 -- 172.21.15.78:0/2708695857 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f607c1579c0 con 0x7f607c06da80 2023-10-14T11:58:32.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.317+0000 7f608404e700 1 -- 172.21.15.78:0/2708695857 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f607c074b80 con 0x7f607c06da80 2023-10-14T11:58:32.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.317+0000 7f60727fc700 1 -- 172.21.15.78:0/2708695857 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6078021030 con 0x7f607c06da80 2023-10-14T11:58:32.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.318+0000 7f60727fc700 1 --2- 172.21.15.78:0/2708695857 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f606006af00 0x7f606006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:32.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.318+0000 7f60727fc700 1 -- 172.21.15.78:0/2708695857 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f607808a8c0 con 0x7f607c06da80 2023-10-14T11:58:32.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.318+0000 7f60815e9700 1 --2- 172.21.15.78:0/2708695857 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f606006af00 0x7f606006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:32.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.318+0000 7f60815e9700 1 --2- 172.21.15.78:0/2708695857 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f606006af00 0x7f606006d3c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f607c151380 tx=0x7f606c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:32.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.322+0000 7f60727fc700 1 -- 172.21.15.78:0/2708695857 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6078055980 con 0x7f607c06da80 2023-10-14T11:58:32.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.554+0000 7f608404e700 1 -- 172.21.15.78:0/2708695857 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f607c153610 con 0x7f607c06da80 2023-10-14T11:58:32.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.554+0000 7f60727fc700 1 -- 172.21.15.78:0/2708695857 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f6078058fa0 con 0x7f607c06da80 2023-10-14T11:58:32.557 INFO:teuthology.orchestra.run.smithi078.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-14T11:58:32.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.556+0000 7f608404e700 1 -- 172.21.15.78:0/2708695857 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f606006af00 msgr2=0x7f606006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:32.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.556+0000 7f608404e700 1 --2- 172.21.15.78:0/2708695857 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f606006af00 0x7f606006d3c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f607c151380 tx=0x7f606c009040 comp rx=0 tx=0).stop 2023-10-14T11:58:32.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.556+0000 7f608404e700 1 -- 172.21.15.78:0/2708695857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f607c06da80 msgr2=0x7f607c1530d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:32.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.556+0000 7f608404e700 1 --2- 172.21.15.78:0/2708695857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f607c06da80 0x7f607c1530d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6078000c00 tx=0x7f6078004750 comp rx=0 tx=0).stop 2023-10-14T11:58:32.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.556+0000 7f608404e700 1 -- 172.21.15.78:0/2708695857 shutdown_connections 2023-10-14T11:58:32.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.556+0000 7f608404e700 1 --2- 172.21.15.78:0/2708695857 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f606006af00 0x7f606006d3c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:32.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.556+0000 7f608404e700 1 --2- 172.21.15.78:0/2708695857 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f607c0ac1d0 0x7f607c1457c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:32.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.556+0000 7f608404e700 1 --2- 172.21.15.78:0/2708695857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f607c06da80 0x7f607c1530d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:32.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.556+0000 7f608404e700 1 --2- 172.21.15.78:0/2708695857 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f607c06de60 0x7f607c145280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:32.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.556+0000 7f608404e700 1 -- 172.21.15.78:0/2708695857 >> 172.21.15.78:0/2708695857 conn(0x7f607c0a8410 msgr2=0x7f607c0c4620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:32.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.556+0000 7f608404e700 1 -- 172.21.15.78:0/2708695857 shutdown_connections 2023-10-14T11:58:32.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:32.556+0000 7f608404e700 1 -- 172.21.15.78:0/2708695857 wait complete. 2023-10-14T11:58:32.570 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-14T11:58:32.570 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-14T11:58:32.571 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi195.front.sepia.ceph.com 2023-10-14T11:58:32.571 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-10-14T11:58:32.571 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-10-14T11:58:32.571 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-14T11:58:32.572 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-14T11:58:32.572 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-14T11:58:32.573 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2023-10-14T11:58:32.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:32.573 DEBUG:teuthology.orchestra.run.smithi195:> ip addr 2023-10-14T11:58:32.594 INFO:teuthology.orchestra.run.smithi195.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-14T11:58:32.595 INFO:teuthology.orchestra.run.smithi195.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-14T11:58:32.595 INFO:teuthology.orchestra.run.smithi195.stdout: inet 127.0.0.1/8 scope host lo 2023-10-14T11:58:32.595 INFO:teuthology.orchestra.run.smithi195.stdout: valid_lft forever preferred_lft forever 2023-10-14T11:58:32.595 INFO:teuthology.orchestra.run.smithi195.stdout: inet6 ::1/128 scope host 2023-10-14T11:58:32.596 INFO:teuthology.orchestra.run.smithi195.stdout: valid_lft forever preferred_lft forever 2023-10-14T11:58:32.596 INFO:teuthology.orchestra.run.smithi195.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T11:58:32.596 INFO:teuthology.orchestra.run.smithi195.stdout: link/ether 0c:c4:7a:d9:36:f0 brd ff:ff:ff:ff:ff:ff 2023-10-14T11:58:32.596 INFO:teuthology.orchestra.run.smithi195.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T11:58:32.596 INFO:teuthology.orchestra.run.smithi195.stdout: link/ether 0c:c4:7a:d9:36:f1 brd ff:ff:ff:ff:ff:ff 2023-10-14T11:58:32.597 INFO:teuthology.orchestra.run.smithi195.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T11:58:32.597 INFO:teuthology.orchestra.run.smithi195.stdout: link/ether 0c:c4:7a:88:80:80 brd ff:ff:ff:ff:ff:ff 2023-10-14T11:58:32.597 INFO:teuthology.orchestra.run.smithi195.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-14T11:58:32.597 INFO:teuthology.orchestra.run.smithi195.stdout: link/ether 0c:c4:7a:88:80:81 brd ff:ff:ff:ff:ff:ff 2023-10-14T11:58:32.598 INFO:teuthology.orchestra.run.smithi195.stdout: inet 172.21.15.195/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-14T11:58:32.598 INFO:teuthology.orchestra.run.smithi195.stdout: valid_lft 42424sec preferred_lft 42424sec 2023-10-14T11:58:32.598 INFO:teuthology.orchestra.run.smithi195.stdout: inet6 fe80::4:bbff:a3de:e49d/64 scope link noprefixroute 2023-10-14T11:58:32.598 INFO:teuthology.orchestra.run.smithi195.stdout: valid_lft forever preferred_lft forever 2023-10-14T11:58:32.599 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-14T11:58:32.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:32.599 DEBUG:teuthology.orchestra.run.smithi195:> (cd / && exec bash -c ' 2023-10-14T11:58:32.600 DEBUG:teuthology.orchestra.run.smithi195:> set -e 2023-10-14T11:58:32.600 DEBUG:teuthology.orchestra.run.smithi195:> sudo ip link add name ceph-brx type bridge 2023-10-14T11:58:32.600 DEBUG:teuthology.orchestra.run.smithi195:> sudo ip addr flush dev ceph-brx 2023-10-14T11:58:32.600 DEBUG:teuthology.orchestra.run.smithi195:> sudo ip link set ceph-brx up 2023-10-14T11:58:32.601 DEBUG:teuthology.orchestra.run.smithi195:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-14T11:58:32.601 DEBUG:teuthology.orchestra.run.smithi195:> ') 2023-10-14T11:58:32.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:32.771 DEBUG:teuthology.orchestra.run.smithi195:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-14T11:58:32.800 INFO:teuthology.orchestra.run.smithi195.stdout:1 2023-10-14T11:58:32.801 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:32.802 DEBUG:teuthology.orchestra.run.smithi195:> route 2023-10-14T11:58:32.878 INFO:teuthology.orchestra.run.smithi195.stdout:Kernel IP routing table 2023-10-14T11:58:32.879 INFO:teuthology.orchestra.run.smithi195.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-14T11:58:32.879 INFO:teuthology.orchestra.run.smithi195.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-14T11:58:32.879 INFO:teuthology.orchestra.run.smithi195.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-14T11:58:32.879 INFO:teuthology.orchestra.run.smithi195.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-14T11:58:32.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:32.880 DEBUG:teuthology.orchestra.run.smithi195:> (cd / && exec bash -c ' 2023-10-14T11:58:32.881 DEBUG:teuthology.orchestra.run.smithi195:> set -e 2023-10-14T11:58:32.881 DEBUG:teuthology.orchestra.run.smithi195:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-14T11:58:32.881 DEBUG:teuthology.orchestra.run.smithi195:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-14T11:58:32.881 DEBUG:teuthology.orchestra.run.smithi195:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-14T11:58:32.882 DEBUG:teuthology.orchestra.run.smithi195:> ') 2023-10-14T11:58:33.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:33.109 DEBUG:teuthology.orchestra.run.smithi195:> ip netns list 2023-10-14T11:58:33.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:33.126 DEBUG:teuthology.orchestra.run.smithi195:> ip netns list-id 2023-10-14T11:58:33.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:33.182 DEBUG:teuthology.orchestra.run.smithi195:> (cd / && exec bash -c ' 2023-10-14T11:58:33.182 DEBUG:teuthology.orchestra.run.smithi195:> set -e 2023-10-14T11:58:33.183 DEBUG:teuthology.orchestra.run.smithi195:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-14T11:58:33.183 DEBUG:teuthology.orchestra.run.smithi195:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2023-10-14T11:58:33.183 DEBUG:teuthology.orchestra.run.smithi195:> ') 2023-10-14T11:58:33.276 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2023-10-14T11:58:33.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:33.277 DEBUG:teuthology.orchestra.run.smithi195:> (cd / && exec bash -c ' 2023-10-14T11:58:33.277 DEBUG:teuthology.orchestra.run.smithi195:> set -e 2023-10-14T11:58:33.278 DEBUG:teuthology.orchestra.run.smithi195:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2023-10-14T11:58:33.278 DEBUG:teuthology.orchestra.run.smithi195:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-14T11:58:33.278 DEBUG:teuthology.orchestra.run.smithi195:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2023-10-14T11:58:33.279 DEBUG:teuthology.orchestra.run.smithi195:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2023-10-14T11:58:33.279 DEBUG:teuthology.orchestra.run.smithi195:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2023-10-14T11:58:33.279 DEBUG:teuthology.orchestra.run.smithi195:> ') 2023-10-14T11:58:33.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:33.488 DEBUG:teuthology.orchestra.run.smithi195:> (cd / && exec bash -c ' 2023-10-14T11:58:33.489 DEBUG:teuthology.orchestra.run.smithi195:> set -e 2023-10-14T11:58:33.489 DEBUG:teuthology.orchestra.run.smithi195:> sudo ip link set brx.0 up 2023-10-14T11:58:33.489 DEBUG:teuthology.orchestra.run.smithi195:> sudo ip link set dev brx.0 master ceph-brx 2023-10-14T11:58:33.490 DEBUG:teuthology.orchestra.run.smithi195:> ') 2023-10-14T11:58:33.590 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T11:58:33.590 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-10-14T11:58:33.610 INFO:teuthology.orchestra.run.smithi195.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-10-14T11:58:33.611 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T11:58:33.611 DEBUG:teuthology.orchestra.run.smithi195:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-10-14T11:58:33.666 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2023-10-14T11:58:33.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:33.667 DEBUG:teuthology.orchestra.run.smithi195:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-14T11:58:33.759 INFO:teuthology.orchestra.run.smithi195.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-14T11:58:33.759 INFO:teuthology.orchestra.run.smithi195.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-14T11:58:33.890 INFO:teuthology.orchestra.run.smithi195.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-14T11:58:33.890 INFO:teuthology.orchestra.run.smithi195.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-14T11:58:33.890 INFO:teuthology.orchestra.run.smithi195.stdout:mount.ceph: resolved to: "172.21.15.78:3300,172.21.15.195:3300,172.21.15.195:3301" 2023-10-14T11:58:33.890 INFO:teuthology.orchestra.run.smithi195.stdout:mount.ceph: trying mount with new device syntax: 1@4c5dc134-3092-406a-a1e1-13e6d9e926e8.cephfs=/ 2023-10-14T11:58:33.891 INFO:teuthology.orchestra.run.smithi195.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.78:3300/172.21.15.195:3300/172.21.15.195:3301" will pass to kernel 2023-10-14T11:58:33.891 INFO:teuthology.orchestra.run.smithi195.stdout:mount.ceph: v2 mount support check returned 2 2023-10-14T11:58:33.891 INFO:teuthology.orchestra.run.smithi195.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-14T11:58:33.891 INFO:teuthology.orchestra.run.smithi195.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T11:58:33.921 INFO:teuthology.orchestra.run.smithi195.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-14T11:58:33.921 INFO:teuthology.orchestra.run.smithi195.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-14T11:58:33.921 INFO:teuthology.orchestra.run.smithi195.stdout:mount.ceph: resolved to: "172.21.15.78:3300,172.21.15.195:3300,172.21.15.195:3301" 2023-10-14T11:58:33.921 INFO:teuthology.orchestra.run.smithi195.stdout:mount.ceph: trying mount with new device syntax: 1@4c5dc134-3092-406a-a1e1-13e6d9e926e8.cephfs=/ 2023-10-14T11:58:33.922 INFO:teuthology.orchestra.run.smithi195.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.78:3300/172.21.15.195:3300/172.21.15.195:3301" will pass to kernel 2023-10-14T11:58:33.922 INFO:teuthology.orchestra.run.smithi195.stdout:mount.ceph: v2 mount support check returned 2 2023-10-14T11:58:33.922 INFO:teuthology.orchestra.run.smithi195.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-14T11:58:33.922 INFO:teuthology.orchestra.run.smithi195.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T11:58:33.922 INFO:teuthology.orchestra.run.smithi195.stdout:mount.ceph: resolved to: "172.21.15.78:6789,172.21.15.195:6789,172.21.15.195:6790" 2023-10-14T11:58:33.923 INFO:teuthology.orchestra.run.smithi195.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.78:6789,172.21.15.195:6789,172.21.15.195:6790:/ 2023-10-14T11:58:33.923 INFO:teuthology.orchestra.run.smithi195.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,key=1,mds_namespace=cephfs,fsid=4c5dc134-3092-406a-a1e1-13e6d9e926e8" will pass to kernel 2023-10-14T11:58:33.939 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-14T11:58:33.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:33.940 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-10-14T11:58:33.968 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:33.969 DEBUG:teuthology.orchestra.run.smithi195:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-14T11:58:34.041 INFO:teuthology.orchestra.run.smithi195.stdout:client4481 2023-10-14T11:58:34.042 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-14T11:58:34.042 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T11:58:34.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:34.099 DEBUG:teuthology.orchestra.run.smithi195:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-14T11:58:34.171 INFO:teuthology.orchestra.run.smithi195.stdout:client4481 2023-10-14T11:58:34.171 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:34.172 DEBUG:teuthology.orchestra.run.smithi195:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-14T11:58:34.242 INFO:teuthology.orchestra.run.smithi195.stdout:client4481 2023-10-14T11:58:34.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:34.243 DEBUG:teuthology.orchestra.run.smithi195:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/4c5dc134-3092-406a-a1e1-13e6d9e926e8.client4481/status') 2023-10-14T11:58:34.323 INFO:teuthology.orchestra.run.smithi195.stdout:instance: client.4481 (3)192.168.0.1:0/3297860199 2023-10-14T11:58:34.323 INFO:teuthology.orchestra.run.smithi195.stdout:blocklisted: false 2023-10-14T11:58:34.324 INFO:teuthology.orchestra.run.smithi195.stderr:0+1 records in 2023-10-14T11:58:34.324 INFO:teuthology.orchestra.run.smithi195.stderr:0+1 records out 2023-10-14T11:58:34.325 INFO:teuthology.orchestra.run.smithi195.stderr:69 bytes copied, 2.0095e-05 s, 3.4 MB/s 2023-10-14T11:58:34.325 INFO:teuthology.run_tasks:Running task exec... 2023-10-14T11:58:34.335 INFO:teuthology.task.exec:Executing custom commands... 2023-10-14T11:58:34.336 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2023-10-14T11:58:34.336 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/crash/warn_recent_interval 0' 2023-10-14T11:58:34.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.472+0000 7fe581718700 1 Processor -- start 2023-10-14T11:58:34.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.472+0000 7fe581718700 1 -- start start 2023-10-14T11:58:34.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.473+0000 7fe581718700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe57c06bb10 0x7fe57c0c0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:34.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.473+0000 7fe581718700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe57c06db10 0x7fe57c14ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:34.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.473+0000 7fe581718700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe57c06d680 con 0x7fe57c1522d0 2023-10-14T11:58:34.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.473+0000 7fe581718700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe57c0c5bc0 con 0x7fe57c06bb10 2023-10-14T11:58:34.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.473+0000 7fe581718700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe57c06e860 con 0x7fe57c06db10 2023-10-14T11:58:34.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.473+0000 7fe57affd700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe57c06bb10 0x7fe57c0c0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:34.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.473+0000 7fe573fff700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fe57c1522d0 0x7fe57c0c0f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50120/0 (socket says 172.21.15.78:50120) 2023-10-14T11:58:34.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.473+0000 7fe57affd700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe57c06bb10 0x7fe57c0c0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45242/0 (socket says 172.21.15.78:45242) 2023-10-14T11:58:34.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.473+0000 7fe573fff700 1 -- 172.21.15.78:0/168866020 learned_addr learned my addr 172.21.15.78:0/168866020 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:34.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.473+0000 7fe57b7fe700 1 --2- 172.21.15.78:0/168866020 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe57c06db10 0x7fe57c14ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:34.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.473+0000 7fe57affd700 1 -- 172.21.15.78:0/168866020 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe57c06db10 msgr2=0x7fe57c14ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:34.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.473+0000 7fe57affd700 1 --2- 172.21.15.78:0/168866020 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe57c06db10 0x7fe57c14ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:34.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.473+0000 7fe57affd700 1 -- 172.21.15.78:0/168866020 >> v1:172.21.15.78:6789/0 conn(0x7fe57c1522d0 legacy=0x7fe57c0c0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:34.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.473+0000 7fe57a7fc700 1 -- 172.21.15.78:0/168866020 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1903901916 0 0) 0x7fe57c06d680 con 0x7fe57c1522d0 2023-10-14T11:58:34.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.473+0000 7fe57affd700 1 -- 172.21.15.78:0/168866020 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe57c152750 con 0x7fe57c06bb10 2023-10-14T11:58:34.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.473+0000 7fe57b7fe700 1 --2- 172.21.15.78:0/168866020 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe57c06db10 0x7fe57c14ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:34.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.474+0000 7fe57affd700 1 --2- 172.21.15.78:0/168866020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe57c06bb10 0x7fe57c0c0900 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe56c009840 tx=0x7fe56c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6afc771c1e1c589 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:34.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.474+0000 7fe57a7fc700 1 -- 172.21.15.78:0/168866020 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56c011830 con 0x7fe57c06bb10 2023-10-14T11:58:34.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.474+0000 7fe57a7fc700 1 -- 172.21.15.78:0/168866020 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe56c011990 con 0x7fe57c06bb10 2023-10-14T11:58:34.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.474+0000 7fe57a7fc700 1 -- 172.21.15.78:0/168866020 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56c011d20 con 0x7fe57c06bb10 2023-10-14T11:58:34.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.474+0000 7fe581718700 1 -- 172.21.15.78:0/168866020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe57c06bb10 msgr2=0x7fe57c0c0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:34.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.474+0000 7fe581718700 1 --2- 172.21.15.78:0/168866020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe57c06bb10 0x7fe57c0c0900 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe56c009840 tx=0x7fe56c009b50 comp rx=0 tx=0).stop 2023-10-14T11:58:34.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.474+0000 7fe581718700 1 -- 172.21.15.78:0/168866020 shutdown_connections 2023-10-14T11:58:34.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.474+0000 7fe581718700 1 --2- 172.21.15.78:0/168866020 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe57c06db10 0x7fe57c14ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:34.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.474+0000 7fe581718700 1 --2- 172.21.15.78:0/168866020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe57c06bb10 0x7fe57c0c0900 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:34.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.474+0000 7fe581718700 1 -- 172.21.15.78:0/168866020 >> 172.21.15.78:0/168866020 conn(0x7fe57c0c3070 msgr2=0x7fe57c0c3480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:34.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.474+0000 7fe581718700 1 -- 172.21.15.78:0/168866020 shutdown_connections 2023-10-14T11:58:34.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.474+0000 7fe581718700 1 -- 172.21.15.78:0/168866020 wait complete. 2023-10-14T11:58:34.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.476+0000 7fe581718700 1 Processor -- start 2023-10-14T11:58:34.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.476+0000 7fe581718700 1 -- start start 2023-10-14T11:58:34.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.477+0000 7fe581718700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe57c06bb10 0x7fe57c1e7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:34.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.477+0000 7fe581718700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe57c06db10 0x7fe57c072340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:34.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.477+0000 7fe581718700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe57c1522d0 0x7fe57c0bf500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:34.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.477+0000 7fe581718700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57c0bc5f0 con 0x7fe57c06db10 2023-10-14T11:58:34.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.477+0000 7fe581718700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57c154070 con 0x7fe57c06bb10 2023-10-14T11:58:34.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.477+0000 7fe581718700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe57c1e8490 con 0x7fe57c1522d0 2023-10-14T11:58:34.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.477+0000 7fe573fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe57c06db10 0x7fe57c072340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:34.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.477+0000 7fe573fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe57c06db10 0x7fe57c072340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45250/0 (socket says 172.21.15.78:45250) 2023-10-14T11:58:34.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.477+0000 7fe573fff700 1 -- 172.21.15.78:0/653590799 learned_addr learned my addr 172.21.15.78:0/653590799 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:34.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.477+0000 7fe57b7fe700 1 --2- 172.21.15.78:0/653590799 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe57c1522d0 0x7fe57c0bf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:34.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.477+0000 7fe57affd700 1 --2- 172.21.15.78:0/653590799 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe57c06bb10 0x7fe57c1e7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:34.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.477+0000 7fe573fff700 1 -- 172.21.15.78:0/653590799 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe57c1522d0 msgr2=0x7fe57c0bf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:34.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.477+0000 7fe573fff700 1 --2- 172.21.15.78:0/653590799 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe57c1522d0 0x7fe57c0bf500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:34.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.477+0000 7fe573fff700 1 -- 172.21.15.78:0/653590799 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe57c06bb10 msgr2=0x7fe57c1e7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:34.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.477+0000 7fe573fff700 1 --2- 172.21.15.78:0/653590799 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe57c06bb10 0x7fe57c1e7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:34.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.477+0000 7fe573fff700 1 -- 172.21.15.78:0/653590799 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe56c0094a0 con 0x7fe57c06db10 2023-10-14T11:58:34.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.477+0000 7fe573fff700 1 --2- 172.21.15.78:0/653590799 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe57c06db10 0x7fe57c072340 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe56400c520 tx=0x7fe56400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:34.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.478+0000 7fe5737fe700 1 -- 172.21.15.78:0/653590799 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe564004c80 con 0x7fe57c06db10 2023-10-14T11:58:34.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.478+0000 7fe5737fe700 1 -- 172.21.15.78:0/653590799 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe564004de0 con 0x7fe57c06db10 2023-10-14T11:58:34.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.478+0000 7fe581718700 1 -- 172.21.15.78:0/653590799 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe57c151b10 con 0x7fe57c06db10 2023-10-14T11:58:34.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.478+0000 7fe5737fe700 1 -- 172.21.15.78:0/653590799 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5640072d0 con 0x7fe57c06db10 2023-10-14T11:58:34.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.478+0000 7fe581718700 1 -- 172.21.15.78:0/653590799 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe57c072880 con 0x7fe57c06db10 2023-10-14T11:58:34.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.479+0000 7fe581718700 1 -- 172.21.15.78:0/653590799 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe57c0ac5a0 con 0x7fe57c06db10 2023-10-14T11:58:34.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.479+0000 7fe5737fe700 1 -- 172.21.15.78:0/653590799 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe564007430 con 0x7fe57c06db10 2023-10-14T11:58:34.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.480+0000 7fe5737fe700 1 --2- 172.21.15.78:0/653590799 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe55c06af00 0x7fe55c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:34.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.480+0000 7fe5737fe700 1 -- 172.21.15.78:0/653590799 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe564089130 con 0x7fe57c06db10 2023-10-14T11:58:34.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.480+0000 7fe57affd700 1 --2- 172.21.15.78:0/653590799 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe55c06af00 0x7fe55c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:34.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.480+0000 7fe57affd700 1 --2- 172.21.15.78:0/653590799 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe55c06af00 0x7fe55c06d3c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe56c0090b0 tx=0x7fe56c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:34.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.484+0000 7fe5737fe700 1 -- 172.21.15.78:0/653590799 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe5640541d0 con 0x7fe57c06db10 2023-10-14T11:58:34.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.676+0000 7fe581718700 1 -- 172.21.15.78:0/653590799 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mgr/crash/warn_recent_interval}] v 0) v1 -- 0x7fe57c074df0 con 0x7fe57c06db10 2023-10-14T11:58:34.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.680+0000 7fe5737fe700 1 -- 172.21.15.78:0/653590799 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/crash/warn_recent_interval}]=0 v10)=0 v10) v1 ==== 128+0+0 (secure 0 0 0) 0x7fe5640577f0 con 0x7fe57c06db10 2023-10-14T11:58:34.684 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.682+0000 7fe581718700 1 -- 172.21.15.78:0/653590799 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe55c06af00 msgr2=0x7fe55c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:34.684 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.682+0000 7fe581718700 1 --2- 172.21.15.78:0/653590799 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe55c06af00 0x7fe55c06d3c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe56c0090b0 tx=0x7fe56c008030 comp rx=0 tx=0).stop 2023-10-14T11:58:34.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.682+0000 7fe581718700 1 -- 172.21.15.78:0/653590799 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe57c06db10 msgr2=0x7fe57c072340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:34.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.682+0000 7fe581718700 1 --2- 172.21.15.78:0/653590799 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe57c06db10 0x7fe57c072340 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe56400c520 tx=0x7fe56400c830 comp rx=0 tx=0).stop 2023-10-14T11:58:34.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.682+0000 7fe581718700 1 -- 172.21.15.78:0/653590799 shutdown_connections 2023-10-14T11:58:34.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.682+0000 7fe581718700 1 --2- 172.21.15.78:0/653590799 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe57c1522d0 0x7fe57c0bf500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:34.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.682+0000 7fe581718700 1 --2- 172.21.15.78:0/653590799 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe55c06af00 0x7fe55c06d3c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:34.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.682+0000 7fe581718700 1 --2- 172.21.15.78:0/653590799 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe57c06bb10 0x7fe57c1e7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:34.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.682+0000 7fe581718700 1 --2- 172.21.15.78:0/653590799 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe57c06db10 0x7fe57c072340 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:34.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.682+0000 7fe581718700 1 -- 172.21.15.78:0/653590799 >> 172.21.15.78:0/653590799 conn(0x7fe57c0c3070 msgr2=0x7fe57c0c1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:34.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.682+0000 7fe581718700 1 -- 172.21.15.78:0/653590799 shutdown_connections 2023-10-14T11:58:34.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.682+0000 7fe581718700 1 -- 172.21.15.78:0/653590799 wait complete. 2023-10-14T11:58:34.697 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2023-10-14T11:58:34.721 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T11:58:34.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.840+0000 7fefe4cce700 1 Processor -- start 2023-10-14T11:58:34.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.840+0000 7fefe4cce700 1 -- start start 2023-10-14T11:58:34.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.841+0000 7fefe4cce700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fefe00c7a00 0x7fefe00c33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:34.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.841+0000 7fefe4cce700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefe0156b20 con 0x7fefe00c72a0 2023-10-14T11:58:34.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.841+0000 7fefe4cce700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefe00c7e40 con 0x7fefe00c44c0 2023-10-14T11:58:34.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.841+0000 7fefe4cce700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefe0155f10 con 0x7fefe00c7a00 2023-10-14T11:58:34.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.841+0000 7fefded9d700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fefe00c7a00 0x7fefe00c33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:34.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.841+0000 7fefddd9b700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fefe00c72a0 0x7fefe00c2c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50140/0 (socket says 172.21.15.78:50140) 2023-10-14T11:58:34.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.841+0000 7fefded9d700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fefe00c7a00 0x7fefe00c33a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45266/0 (socket says 172.21.15.78:45266) 2023-10-14T11:58:34.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.841+0000 7fefddd9b700 1 -- 172.21.15.78:0/120416427 learned_addr learned my addr 172.21.15.78:0/120416427 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:34.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.841+0000 7fefded9d700 1 -- 172.21.15.78:0/120416427 >> v1:172.21.15.195:6789/0 conn(0x7fefe00c44c0 legacy=0x7fefe00c86e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:34.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.841+0000 7fefded9d700 1 -- 172.21.15.78:0/120416427 >> v1:172.21.15.78:6789/0 conn(0x7fefe00c72a0 legacy=0x7fefe00c2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:34.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.841+0000 7fefded9d700 1 -- 172.21.15.78:0/120416427 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe01561f0 con 0x7fefe00c7a00 2023-10-14T11:58:34.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.841+0000 7fefdd59a700 1 -- 172.21.15.78:0/120416427 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2351459744 0 0) 0x7fefe0156b20 con 0x7fefe00c72a0 2023-10-14T11:58:34.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.842+0000 7fefded9d700 1 --2- 172.21.15.78:0/120416427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefe00c7a00 0x7fefe00c33a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fefd8009d90 tx=0x7fefd800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1041c20bd94e8dcc server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:34.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.842+0000 7fefdd59a700 1 -- 172.21.15.78:0/120416427 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd800e070 con 0x7fefe00c7a00 2023-10-14T11:58:34.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.842+0000 7fefdd59a700 1 -- 172.21.15.78:0/120416427 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefd800bb90 con 0x7fefe00c7a00 2023-10-14T11:58:34.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.842+0000 7fefdd59a700 1 -- 172.21.15.78:0/120416427 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd8003a20 con 0x7fefe00c7a00 2023-10-14T11:58:34.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.842+0000 7fefe4cce700 1 -- 172.21.15.78:0/120416427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefe00c7a00 msgr2=0x7fefe00c33a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:34.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.842+0000 7fefe4cce700 1 --2- 172.21.15.78:0/120416427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefe00c7a00 0x7fefe00c33a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fefd8009d90 tx=0x7fefd800b330 comp rx=0 tx=0).stop 2023-10-14T11:58:34.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.842+0000 7fefe4cce700 1 -- 172.21.15.78:0/120416427 shutdown_connections 2023-10-14T11:58:34.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.842+0000 7fefe4cce700 1 --2- 172.21.15.78:0/120416427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefe00c7a00 0x7fefe00c33a0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:34.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.842+0000 7fefe4cce700 1 -- 172.21.15.78:0/120416427 >> 172.21.15.78:0/120416427 conn(0x7fefe00737d0 msgr2=0x7fefe0073be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:34.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.842+0000 7fefe4cce700 1 -- 172.21.15.78:0/120416427 shutdown_connections 2023-10-14T11:58:34.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.842+0000 7fefe4cce700 1 -- 172.21.15.78:0/120416427 wait complete. 2023-10-14T11:58:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.844+0000 7fefe4cce700 1 Processor -- start 2023-10-14T11:58:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.844+0000 7fefe4cce700 1 -- start start 2023-10-14T11:58:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.844+0000 7fefe4cce700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fefe00c44c0 0x7fefe00c4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.844+0000 7fefe4cce700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fefe00c72a0 0x7fefe014ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:34.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.844+0000 7fefe4cce700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefe00c7a00 0x7fefe014f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:34.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.844+0000 7fefe4cce700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe00ab5b0 con 0x7fefe00c7a00 2023-10-14T11:58:34.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.844+0000 7fefe4cce700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe00ab890 con 0x7fefe00c72a0 2023-10-14T11:58:34.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.844+0000 7fefe4cce700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefe00abb70 con 0x7fefe00c44c0 2023-10-14T11:58:34.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.844+0000 7fefded9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefe00c7a00 0x7fefe014f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:34.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.844+0000 7fefded9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefe00c7a00 0x7fefe014f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45270/0 (socket says 172.21.15.78:45270) 2023-10-14T11:58:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.844+0000 7fefded9d700 1 -- 172.21.15.78:0/3360503708 learned_addr learned my addr 172.21.15.78:0/3360503708 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.844+0000 7fefddd9b700 1 --2- 172.21.15.78:0/3360503708 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fefe00c72a0 0x7fefe014ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.844+0000 7fefde59c700 1 --2- 172.21.15.78:0/3360503708 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fefe00c44c0 0x7fefe00c4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.845+0000 7fefded9d700 1 -- 172.21.15.78:0/3360503708 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fefe00c44c0 msgr2=0x7fefe00c4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.845+0000 7fefded9d700 1 --2- 172.21.15.78:0/3360503708 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fefe00c44c0 0x7fefe00c4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.845+0000 7fefded9d700 1 -- 172.21.15.78:0/3360503708 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fefe00c72a0 msgr2=0x7fefe014ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.845+0000 7fefded9d700 1 --2- 172.21.15.78:0/3360503708 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fefe00c72a0 0x7fefe014ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.845+0000 7fefded9d700 1 -- 172.21.15.78:0/3360503708 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd8009a70 con 0x7fefe00c7a00 2023-10-14T11:58:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.845+0000 7fefddd9b700 1 --2- 172.21.15.78:0/3360503708 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fefe00c72a0 0x7fefe014ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.845+0000 7fefde59c700 1 --2- 172.21.15.78:0/3360503708 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fefe00c44c0 0x7fefe00c4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.845+0000 7fefded9d700 1 --2- 172.21.15.78:0/3360503708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefe00c7a00 0x7fefe014f3a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fefe00c5ce0 tx=0x7fefd8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.845+0000 7fefceffd700 1 -- 172.21.15.78:0/3360503708 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd8012630 con 0x7fefe00c7a00 2023-10-14T11:58:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.845+0000 7fefceffd700 1 -- 172.21.15.78:0/3360503708 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefd80127b0 con 0x7fefe00c7a00 2023-10-14T11:58:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.845+0000 7fefe4cce700 1 -- 172.21.15.78:0/3360503708 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefe01ed390 con 0x7fefe00c7a00 2023-10-14T11:58:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.845+0000 7fefceffd700 1 -- 172.21.15.78:0/3360503708 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd8012b60 con 0x7fefe00c7a00 2023-10-14T11:58:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.845+0000 7fefe4cce700 1 -- 172.21.15.78:0/3360503708 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefe00c5320 con 0x7fefe00c7a00 2023-10-14T11:58:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.846+0000 7fefe4cce700 1 -- 172.21.15.78:0/3360503708 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefe00ab6f0 con 0x7fefe00c7a00 2023-10-14T11:58:34.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.846+0000 7fefceffd700 1 -- 172.21.15.78:0/3360503708 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fefd8012cc0 con 0x7fefe00c7a00 2023-10-14T11:58:34.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.847+0000 7fefceffd700 1 --2- 172.21.15.78:0/3360503708 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fefc006af00 0x7fefc006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:34.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.847+0000 7fefceffd700 1 -- 172.21.15.78:0/3360503708 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fefd808b9e0 con 0x7fefe00c7a00 2023-10-14T11:58:34.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.847+0000 7fefde59c700 1 --2- 172.21.15.78:0/3360503708 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fefc006af00 0x7fefc006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:34.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.847+0000 7fefde59c700 1 --2- 172.21.15.78:0/3360503708 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fefc006af00 0x7fefc006d3c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fefd4009930 tx=0x7fefd4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:34.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:34.851+0000 7fefceffd700 1 -- 172.21.15.78:0/3360503708 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fefe00ab6f0 con 0x7fefe00c7a00 2023-10-14T11:58:35.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.041+0000 7fefe4cce700 1 -- 172.21.15.78:0/3360503708 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fefe00ab6f0 con 0x7fefe00c7a00 2023-10-14T11:58:35.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.042+0000 7fefceffd700 1 -- 172.21.15.78:0/3360503708 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16481 (secure 0 0 0) 0x7fefe00ab6f0 con 0x7fefe00c7a00 2023-10-14T11:58:35.045 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:35.045 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":21,"fsid":"4c5dc134-3092-406a-a1e1-13e6d9e926e8","created":"2023-10-14T11:57:57.363750+0000","modified":"2023-10-14T11:58:16.139413+0000","last_up_change":"2023-10-14T11:58:06.982472+0000","last_in_change":"2023-10-14T11:57:59.821373+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T11:58:09.001200+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T11:58:12.778573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T11:58:13.490286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"625a5fc0-9474-4eae-b80e-14c27a74ad19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6801","nonce":3884969780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6803","nonce":3884969780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6807","nonce":3884969780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6805","nonce":3884969780}]},"public_addr":"172.21.15.78:6801/3884969780","cluster_addr":"172.21.15.78:6803/3884969780","heartbeat_back_addr":"172.21.15.78:6807/3884969780","heartbeat_front_addr":"172.21.15.78:6805/3884969780","state":["exists","up"]},{"osd":1,"uuid":"49d63db1-2981-49ab-9c40-c50fc744b982","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6809","nonce":2176928406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6811","nonce":2176928406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6815","nonce":2176928406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6813","nonce":2176928406}]},"public_addr":"172.21.15.78:6809/2176928406","cluster_addr":"172.21.15.78:6811/2176928406","heartbeat_back_addr":"172.21.15.78:6815/2176928406","heartbeat_front_addr":"172.21.15.78:6813/2176928406","state":["exists","up"]},{"osd":2,"uuid":"1d7d0b29-cbd1-4717-9fe6-765a8d4998b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6825","nonce":3647044577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6827","nonce":3647044577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6831","nonce":3647044577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6829","nonce":3647044577}]},"public_addr":"172.21.15.78:6825/3647044577","cluster_addr":"172.21.15.78:6827/3647044577","heartbeat_back_addr":"172.21.15.78:6831/3647044577","heartbeat_front_addr":"172.21.15.78:6829/3647044577","state":["exists","up"]},{"osd":3,"uuid":"2f75ded1-e9e2-40e7-99ae-45e1e425e1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6817","nonce":3584112288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6819","nonce":3584112288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6823","nonce":3584112288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6821","nonce":3584112288}]},"public_addr":"172.21.15.78:6817/3584112288","cluster_addr":"172.21.15.78:6819/3584112288","heartbeat_back_addr":"172.21.15.78:6823/3584112288","heartbeat_front_addr":"172.21.15.78:6821/3584112288","state":["exists","up"]},{"osd":4,"uuid":"5130b432-8358-46cb-8a31-3064c976f6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6801","nonce":3732756301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6803","nonce":3732756301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6807","nonce":3732756301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6805","nonce":3732756301}]},"public_addr":"172.21.15.195:6801/3732756301","cluster_addr":"172.21.15.195:6803/3732756301","heartbeat_back_addr":"172.21.15.195:6807/3732756301","heartbeat_front_addr":"172.21.15.195:6805/3732756301","state":["exists","up"]},{"osd":5,"uuid":"60e234a9-a506-4e40-9c19-49a58f3cf28c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6825","nonce":2857706411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6827","nonce":2857706411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6831","nonce":2857706411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6829","nonce":2857706411}]},"public_addr":"172.21.15.195:6825/2857706411","cluster_addr":"172.21.15.195:6827/2857706411","heartbeat_back_addr":"172.21.15.195:6831/2857706411","heartbeat_front_addr":"172.21.15.195:6829/2857706411","state":["exists","up"]},{"osd":6,"uuid":"df7d2dd6-26b6-4eb1-96e5-94d1f3955b33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6810","nonce":72554374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6813","nonce":72554374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6820","nonce":72554374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6817","nonce":72554374}]},"public_addr":"172.21.15.195:6810/72554374","cluster_addr":"172.21.15.195:6813/72554374","heartbeat_back_addr":"172.21.15.195:6820/72554374","heartbeat_front_addr":"172.21.15.195:6817/72554374","state":["exists","up"]},{"osd":7,"uuid":"47a9d4de-f56d-4aa4-9900-c4a9cf9b2f97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6809","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6811","nonce":482472979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6816","nonce":482472979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6823","nonce":482472979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6821","nonce":482472979}]},"public_addr":"172.21.15.195:6811/482472979","cluster_addr":"172.21.15.195:6816/482472979","heartbeat_back_addr":"172.21.15.195:6823/482472979","heartbeat_front_addr":"172.21.15.195:6821/482472979","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.754003+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.005994+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.043603+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.024506+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.062585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.033767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.994825+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.038300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T11:58:35.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.044+0000 7fefe4cce700 1 -- 172.21.15.78:0/3360503708 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fefc006af00 msgr2=0x7fefc006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:35.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.044+0000 7fefe4cce700 1 --2- 172.21.15.78:0/3360503708 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fefc006af00 0x7fefc006d3c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fefd4009930 tx=0x7fefd4008040 comp rx=0 tx=0).stop 2023-10-14T11:58:35.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.044+0000 7fefe4cce700 1 -- 172.21.15.78:0/3360503708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefe00c7a00 msgr2=0x7fefe014f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:35.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.044+0000 7fefe4cce700 1 --2- 172.21.15.78:0/3360503708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefe00c7a00 0x7fefe014f3a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fefe00c5ce0 tx=0x7fefd8004ad0 comp rx=0 tx=0).stop 2023-10-14T11:58:35.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.044+0000 7fefe4cce700 1 -- 172.21.15.78:0/3360503708 shutdown_connections 2023-10-14T11:58:35.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.044+0000 7fefe4cce700 1 --2- 172.21.15.78:0/3360503708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefe00c7a00 0x7fefe014f3a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:35.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.044+0000 7fefe4cce700 1 --2- 172.21.15.78:0/3360503708 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fefe00c44c0 0x7fefe00c4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:35.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.044+0000 7fefe4cce700 1 --2- 172.21.15.78:0/3360503708 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fefc006af00 0x7fefc006d3c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:35.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.044+0000 7fefe4cce700 1 --2- 172.21.15.78:0/3360503708 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fefe00c72a0 0x7fefe014ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:35.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.044+0000 7fefe4cce700 1 -- 172.21.15.78:0/3360503708 >> 172.21.15.78:0/3360503708 conn(0x7fefe00737d0 msgr2=0x7fefe0153ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:35.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.044+0000 7fefe4cce700 1 -- 172.21.15.78:0/3360503708 shutdown_connections 2023-10-14T11:58:35.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.044+0000 7fefe4cce700 1 -- 172.21.15.78:0/3360503708 wait complete. 2023-10-14T11:58:35.059 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T11:58:09.001200+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T11:58:12.778573+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T11:58:13.490286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T11:58:35.060 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T11:58:35.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.182+0000 7f8de2e2c700 1 Processor -- start 2023-10-14T11:58:35.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.182+0000 7f8de2e2c700 1 -- start start 2023-10-14T11:58:35.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.182+0000 7f8de2e2c700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8ddc14ed10 0x7f8ddc152ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:35.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.182+0000 7f8de2e2c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f8ddc0c35e0 0x7f8ddc153400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:35.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.182+0000 7f8de2e2c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ddc0c26d0 con 0x7f8ddc153940 2023-10-14T11:58:35.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.182+0000 7f8de2e2c700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ddc0a8520 con 0x7f8ddc14ed10 2023-10-14T11:58:35.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.182+0000 7f8de2e2c700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ddc0c29b0 con 0x7f8ddc0c35e0 2023-10-14T11:58:35.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.183+0000 7f8de0bc8700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8ddc14ed10 0x7f8ddc152ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:35.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.183+0000 7f8de0bc8700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8ddc14ed10 0x7f8ddc152ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45276/0 (socket says 172.21.15.78:45276) 2023-10-14T11:58:35.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.183+0000 7f8de0bc8700 1 -- 172.21.15.78:0/2329077276 learned_addr learned my addr 172.21.15.78:0/2329077276 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:35.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.183+0000 7f8ddbfff700 1 --2- 172.21.15.78:0/2329077276 >> v2:172.21.15.195:3300/0 conn(0x7f8ddc0c35e0 0x7f8ddc153400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:35.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.183+0000 7f8de0bc8700 1 -- 172.21.15.78:0/2329077276 >> v2:172.21.15.195:3300/0 conn(0x7f8ddc0c35e0 msgr2=0x7f8ddc153400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:35.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.183+0000 7f8de0bc8700 1 --2- 172.21.15.78:0/2329077276 >> v2:172.21.15.195:3300/0 conn(0x7f8ddc0c35e0 0x7f8ddc153400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:35.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.183+0000 7f8de0bc8700 1 -- 172.21.15.78:0/2329077276 >> v1:172.21.15.195:6789/0 conn(0x7f8ddc153940 legacy=0x7f8ddc154c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:35.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.183+0000 7f8de0bc8700 1 -- 172.21.15.78:0/2329077276 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ddc0c2c90 con 0x7f8ddc14ed10 2023-10-14T11:58:35.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.183+0000 7f8de0bc8700 1 --2- 172.21.15.78:0/2329077276 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ddc14ed10 0x7f8ddc152ec0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc155d40 tx=0x7f8dd000be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=828096fa86702551 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:35.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.183+0000 7f8ddb7fe700 1 -- 172.21.15.78:0/2329077276 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd000dc70 con 0x7f8ddc14ed10 2023-10-14T11:58:35.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.183+0000 7f8ddb7fe700 1 -- 172.21.15.78:0/2329077276 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dd000ddd0 con 0x7f8ddc14ed10 2023-10-14T11:58:35.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.183+0000 7f8ddb7fe700 1 -- 172.21.15.78:0/2329077276 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd0010690 con 0x7f8ddc14ed10 2023-10-14T11:58:35.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.184+0000 7f8de2e2c700 1 -- 172.21.15.78:0/2329077276 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ddc14ed10 msgr2=0x7f8ddc152ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:35.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.184+0000 7f8de2e2c700 1 --2- 172.21.15.78:0/2329077276 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ddc14ed10 0x7f8ddc152ec0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc155d40 tx=0x7f8dd000be70 comp rx=0 tx=0).stop 2023-10-14T11:58:35.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.184+0000 7f8de2e2c700 1 -- 172.21.15.78:0/2329077276 shutdown_connections 2023-10-14T11:58:35.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.184+0000 7f8de2e2c700 1 --2- 172.21.15.78:0/2329077276 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ddc14ed10 0x7f8ddc152ec0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:35.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.184+0000 7f8de2e2c700 1 --2- 172.21.15.78:0/2329077276 >> v2:172.21.15.195:3300/0 conn(0x7f8ddc0c35e0 0x7f8ddc153400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:35.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.184+0000 7f8de2e2c700 1 -- 172.21.15.78:0/2329077276 >> 172.21.15.78:0/2329077276 conn(0x7f8ddc0c3a50 msgr2=0x7f8ddc0c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:35.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.184+0000 7f8de2e2c700 1 -- 172.21.15.78:0/2329077276 shutdown_connections 2023-10-14T11:58:35.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.184+0000 7f8de2e2c700 1 -- 172.21.15.78:0/2329077276 wait complete. 2023-10-14T11:58:35.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.185+0000 7f8de2e2c700 1 Processor -- start 2023-10-14T11:58:35.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.185+0000 7f8de2e2c700 1 -- start start 2023-10-14T11:58:35.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.186+0000 7f8de2e2c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8ddc14ed10 0x7f8ddc14f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:35.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.186+0000 7f8de2e2c700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8ddc153940 0x7f8ddc0a9c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:35.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.186+0000 7f8de2e2c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ddc0aa170 0x7f8ddc147b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:35.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.186+0000 7f8de2e2c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ddc151460 con 0x7f8ddc0aa170 2023-10-14T11:58:35.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.186+0000 7f8de2e2c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ddc151700 con 0x7f8ddc14ed10 2023-10-14T11:58:35.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.186+0000 7f8de2e2c700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ddc1519e0 con 0x7f8ddc153940 2023-10-14T11:58:35.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.186+0000 7f8de13c9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ddc0aa170 0x7f8ddc147b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:35.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.186+0000 7f8de13c9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ddc0aa170 0x7f8ddc147b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45286/0 (socket says 172.21.15.78:45286) 2023-10-14T11:58:35.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.186+0000 7f8de13c9700 1 -- 172.21.15.78:0/2084424551 learned_addr learned my addr 172.21.15.78:0/2084424551 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:35.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.187+0000 7f8de0bc8700 1 --2- 172.21.15.78:0/2084424551 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8ddc14ed10 0x7f8ddc14f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:35.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.187+0000 7f8ddbfff700 1 --2- 172.21.15.78:0/2084424551 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8ddc153940 0x7f8ddc0a9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:35.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.187+0000 7f8de13c9700 1 -- 172.21.15.78:0/2084424551 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8ddc153940 msgr2=0x7f8ddc0a9c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:35.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.187+0000 7f8de13c9700 1 --2- 172.21.15.78:0/2084424551 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8ddc153940 0x7f8ddc0a9c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:35.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.187+0000 7f8de13c9700 1 -- 172.21.15.78:0/2084424551 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8ddc14ed10 msgr2=0x7f8ddc14f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:35.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.187+0000 7f8de13c9700 1 --2- 172.21.15.78:0/2084424551 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8ddc14ed10 0x7f8ddc14f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:35.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.187+0000 7f8de13c9700 1 -- 172.21.15.78:0/2084424551 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd000b7f0 con 0x7f8ddc0aa170 2023-10-14T11:58:35.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.187+0000 7f8ddbfff700 1 --2- 172.21.15.78:0/2084424551 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8ddc153940 0x7f8ddc0a9c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:35.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.187+0000 7f8de13c9700 1 --2- 172.21.15.78:0/2084424551 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ddc0aa170 0x7f8ddc147b30 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8dd400caa0 tx=0x7f8dd400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:35.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.187+0000 7f8dd97fa700 1 -- 172.21.15.78:0/2084424551 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd4003c00 con 0x7f8ddc0aa170 2023-10-14T11:58:35.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.187+0000 7f8dd97fa700 1 -- 172.21.15.78:0/2084424551 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dd4003d60 con 0x7f8ddc0aa170 2023-10-14T11:58:35.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.187+0000 7f8dd97fa700 1 -- 172.21.15.78:0/2084424551 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd400aac0 con 0x7f8ddc0aa170 2023-10-14T11:58:35.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.187+0000 7f8de2e2c700 1 -- 172.21.15.78:0/2084424551 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ddc151cc0 con 0x7f8ddc0aa170 2023-10-14T11:58:35.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.187+0000 7f8de2e2c700 1 -- 172.21.15.78:0/2084424551 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ddc151fa0 con 0x7f8ddc0aa170 2023-10-14T11:58:35.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.188+0000 7f8de2e2c700 1 -- 172.21.15.78:0/2084424551 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ddc1e87e0 con 0x7f8ddc0aa170 2023-10-14T11:58:35.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.189+0000 7f8dd97fa700 1 -- 172.21.15.78:0/2084424551 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8dd400ac20 con 0x7f8ddc0aa170 2023-10-14T11:58:35.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.189+0000 7f8dd97fa700 1 --2- 172.21.15.78:0/2084424551 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8dc006af00 0x7f8dc006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:35.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.189+0000 7f8dd97fa700 1 -- 172.21.15.78:0/2084424551 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8dd40894f0 con 0x7f8ddc0aa170 2023-10-14T11:58:35.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.189+0000 7f8de0bc8700 1 --2- 172.21.15.78:0/2084424551 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8dc006af00 0x7f8dc006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:35.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.189+0000 7f8de0bc8700 1 --2- 172.21.15.78:0/2084424551 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8dc006af00 0x7f8dc006d3c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8dd000beb0 tx=0x7f8dd0013000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:35.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.193+0000 7f8dd97fa700 1 -- 172.21.15.78:0/2084424551 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8dd4054630 con 0x7f8ddc0aa170 2023-10-14T11:58:35.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.382+0000 7f8de2e2c700 1 -- 172.21.15.78:0/2084424551 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8ddc074b80 con 0x7f8ddc0aa170 2023-10-14T11:58:35.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.383+0000 7f8dd97fa700 1 -- 172.21.15.78:0/2084424551 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8dd4057c50 con 0x7f8ddc0aa170 2023-10-14T11:58:35.385 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-10-14T11:58:35.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.385+0000 7f8de2e2c700 1 -- 172.21.15.78:0/2084424551 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8dc006af00 msgr2=0x7f8dc006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:35.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.385+0000 7f8de2e2c700 1 --2- 172.21.15.78:0/2084424551 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8dc006af00 0x7f8dc006d3c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8dd000beb0 tx=0x7f8dd0013000 comp rx=0 tx=0).stop 2023-10-14T11:58:35.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.385+0000 7f8de2e2c700 1 -- 172.21.15.78:0/2084424551 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ddc0aa170 msgr2=0x7f8ddc147b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:35.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.385+0000 7f8de2e2c700 1 --2- 172.21.15.78:0/2084424551 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ddc0aa170 0x7f8ddc147b30 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8dd400caa0 tx=0x7f8dd400ce60 comp rx=0 tx=0).stop 2023-10-14T11:58:35.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.385+0000 7f8de2e2c700 1 -- 172.21.15.78:0/2084424551 shutdown_connections 2023-10-14T11:58:35.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.385+0000 7f8de2e2c700 1 --2- 172.21.15.78:0/2084424551 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ddc0aa170 0x7f8ddc147b30 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:35.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.385+0000 7f8de2e2c700 1 --2- 172.21.15.78:0/2084424551 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8dc006af00 0x7f8dc006d3c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:35.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.385+0000 7f8de2e2c700 1 --2- 172.21.15.78:0/2084424551 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8ddc14ed10 0x7f8ddc14f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:35.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.385+0000 7f8de2e2c700 1 --2- 172.21.15.78:0/2084424551 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8ddc153940 0x7f8ddc0a9c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:35.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.385+0000 7f8de2e2c700 1 -- 172.21.15.78:0/2084424551 >> 172.21.15.78:0/2084424551 conn(0x7f8ddc0c3a50 msgr2=0x7f8ddc071e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:35.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.385+0000 7f8de2e2c700 1 -- 172.21.15.78:0/2084424551 shutdown_connections 2023-10-14T11:58:35.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.385+0000 7f8de2e2c700 1 -- 172.21.15.78:0/2084424551 wait complete. 2023-10-14T11:58:35.399 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T11:58:35.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.519+0000 7f7c2df0d700 1 Processor -- start 2023-10-14T11:58:35.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.519+0000 7f7c2df0d700 1 -- start start 2023-10-14T11:58:35.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.520+0000 7f7c2df0d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f7c280c35e0 0x7f7c281535d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:35.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.520+0000 7f7c2df0d700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c280c26d0 con 0x7f7c2814ed10 2023-10-14T11:58:35.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.520+0000 7f7c2df0d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c280c29b0 con 0x7f7c28154c20 2023-10-14T11:58:35.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.520+0000 7f7c2df0d700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c280a8520 con 0x7f7c280c35e0 2023-10-14T11:58:35.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.520+0000 7f7c277fe700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f7c2814ed10 0x7f7c28152ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50166/0 (socket says 172.21.15.78:50166) 2023-10-14T11:58:35.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.520+0000 7f7c277fe700 1 -- 172.21.15.78:0/2590560743 learned_addr learned my addr 172.21.15.78:0/2590560743 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:35.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.520+0000 7f7c26ffd700 1 --2- 172.21.15.78:0/2590560743 >> v2:172.21.15.195:3300/0 conn(0x7f7c280c35e0 0x7f7c281535d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:35.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.520+0000 7f7c267fc700 1 -- 172.21.15.78:0/2590560743 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2752199738 0 0) 0x7f7c280c26d0 con 0x7f7c2814ed10 2023-10-14T11:58:35.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.520+0000 7f7c267fc700 1 -- 172.21.15.78:0/2590560743 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c100035b0 con 0x7f7c2814ed10 2023-10-14T11:58:35.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.521+0000 7f7c267fc700 1 -- 172.21.15.78:0/2590560743 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3504485527 0 0) 0x7f7c100035b0 con 0x7f7c2814ed10 2023-10-14T11:58:35.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.521+0000 7f7c267fc700 1 -- 172.21.15.78:0/2590560743 >> v2:172.21.15.195:3300/0 conn(0x7f7c280c35e0 msgr2=0x7f7c281535d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:35.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.521+0000 7f7c267fc700 1 --2- 172.21.15.78:0/2590560743 >> v2:172.21.15.195:3300/0 conn(0x7f7c280c35e0 0x7f7c281535d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:35.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.521+0000 7f7c267fc700 1 -- 172.21.15.78:0/2590560743 >> v1:172.21.15.195:6789/0 conn(0x7f7c28154c20 legacy=0x7f7c28155030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:35.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.521+0000 7f7c267fc700 1 -- 172.21.15.78:0/2590560743 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c280c2c90 con 0x7f7c2814ed10 2023-10-14T11:58:35.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.521+0000 7f7c267fc700 1 -- 172.21.15.78:0/2590560743 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441799917 0 0) 0x7f7c1c001670 con 0x7f7c2814ed10 2023-10-14T11:58:35.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.521+0000 7f7c267fc700 1 -- 172.21.15.78:0/2590560743 >> v1:172.21.15.78:6789/0 conn(0x7f7c2814ed10 legacy=0x7f7c28152ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:35.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.521+0000 7f7c267fc700 1 --2- 172.21.15.78:0/2590560743 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c10003a00 0x7f7c10003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:35.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.521+0000 7f7c267fc700 1 --2- 172.21.15.78:0/2590560743 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7c10004850 0x7f7c10004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:35.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.521+0000 7f7c267fc700 1 --2- 172.21.15.78:0/2590560743 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7c10005210 0x7f7c1000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:35.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.521+0000 7f7c267fc700 1 -- 172.21.15.78:0/2590560743 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c280a8520 con 0x7f7c10003a00 2023-10-14T11:58:35.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.521+0000 7f7c267fc700 1 -- 172.21.15.78:0/2590560743 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1000c1b0 con 0x7f7c10004850 2023-10-14T11:58:35.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.521+0000 7f7c267fc700 1 -- 172.21.15.78:0/2590560743 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1000c370 con 0x7f7c10005210 2023-10-14T11:58:35.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.521+0000 7f7c277fe700 1 --2- 172.21.15.78:0/2590560743 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c10003a00 0x7f7c10003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:35.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.521+0000 7f7c26ffd700 1 --2- 172.21.15.78:0/2590560743 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7c10004850 0x7f7c10004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:35.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.521+0000 7f7c27fff700 1 --2- 172.21.15.78:0/2590560743 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7c10005210 0x7f7c1000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:35.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.521+0000 7f7c277fe700 1 -- 172.21.15.78:0/2590560743 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7c10005210 msgr2=0x7f7c1000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:35.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.521+0000 7f7c277fe700 1 --2- 172.21.15.78:0/2590560743 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7c10005210 0x7f7c1000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:35.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.521+0000 7f7c277fe700 1 -- 172.21.15.78:0/2590560743 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7c10004850 msgr2=0x7f7c10004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:35.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.521+0000 7f7c277fe700 1 --2- 172.21.15.78:0/2590560743 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7c10004850 0x7f7c10004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:35.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.521+0000 7f7c277fe700 1 -- 172.21.15.78:0/2590560743 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c100035b0 con 0x7f7c10003a00 2023-10-14T11:58:35.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.521+0000 7f7c27fff700 1 -- 172.21.15.78:0/2590560743 reap_dead start 2023-10-14T11:58:35.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.521+0000 7f7c277fe700 1 --2- 172.21.15.78:0/2590560743 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c10003a00 0x7f7c10003e20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c00c2a0 tx=0x7f7c10005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c196a2a08ac67af5 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:35.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.521+0000 7f7c267fc700 1 -- 172.21.15.78:0/2590560743 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c1c007c10 con 0x7f7c10003a00 2023-10-14T11:58:35.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.521+0000 7f7c267fc700 1 -- 172.21.15.78:0/2590560743 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c1c00cde0 con 0x7f7c10003a00 2023-10-14T11:58:35.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.521+0000 7f7c267fc700 1 -- 172.21.15.78:0/2590560743 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c1c005ba0 con 0x7f7c10003a00 2023-10-14T11:58:35.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.522+0000 7f7c2df0d700 1 -- 172.21.15.78:0/2590560743 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c10003a00 msgr2=0x7f7c10003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:35.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.522+0000 7f7c2df0d700 1 --2- 172.21.15.78:0/2590560743 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c10003a00 0x7f7c10003e20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c00c2a0 tx=0x7f7c10005880 comp rx=0 tx=0).stop 2023-10-14T11:58:35.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.522+0000 7f7c2df0d700 1 -- 172.21.15.78:0/2590560743 shutdown_connections 2023-10-14T11:58:35.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.522+0000 7f7c2df0d700 1 --2- 172.21.15.78:0/2590560743 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c10003a00 0x7f7c10003e20 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:35.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.522+0000 7f7c2df0d700 1 -- 172.21.15.78:0/2590560743 >> 172.21.15.78:0/2590560743 conn(0x7f7c280c3a50 msgr2=0x7f7c280c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:35.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.522+0000 7f7c2df0d700 1 -- 172.21.15.78:0/2590560743 shutdown_connections 2023-10-14T11:58:35.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.522+0000 7f7c2df0d700 1 -- 172.21.15.78:0/2590560743 wait complete. 2023-10-14T11:58:35.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.523+0000 7f7c2df0d700 1 Processor -- start 2023-10-14T11:58:35.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.523+0000 7f7c2df0d700 1 -- start start 2023-10-14T11:58:35.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.524+0000 7f7c2df0d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c10003a00 0x7f7c281e8390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:35.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.524+0000 7f7c2df0d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7c2814c160 0x7f7c280ab850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:35.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.524+0000 7f7c2df0d700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7c2814f4a0 0x7f7c281eb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:35.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.524+0000 7f7c2df0d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c281e88d0 con 0x7f7c10003a00 2023-10-14T11:58:35.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.524+0000 7f7c2df0d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c281eb120 con 0x7f7c2814c160 2023-10-14T11:58:35.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.524+0000 7f7c2df0d700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2814b260 con 0x7f7c2814f4a0 2023-10-14T11:58:35.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.524+0000 7f7c277fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c10003a00 0x7f7c281e8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:35.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.524+0000 7f7c277fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c10003a00 0x7f7c281e8390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45302/0 (socket says 172.21.15.78:45302) 2023-10-14T11:58:35.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.524+0000 7f7c277fe700 1 -- 172.21.15.78:0/1269339829 learned_addr learned my addr 172.21.15.78:0/1269339829 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:35.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.524+0000 7f7c277fe700 1 -- 172.21.15.78:0/1269339829 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7c2814f4a0 msgr2=0x7f7c281eb5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:35.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.524+0000 7f7c277fe700 1 --2- 172.21.15.78:0/1269339829 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7c2814f4a0 0x7f7c281eb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:35.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.524+0000 7f7c277fe700 1 -- 172.21.15.78:0/1269339829 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7c2814c160 msgr2=0x7f7c280ab850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:35.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.524+0000 7f7c277fe700 1 --2- 172.21.15.78:0/1269339829 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7c2814c160 0x7f7c280ab850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:35.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.524+0000 7f7c277fe700 1 -- 172.21.15.78:0/1269339829 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c2814b540 con 0x7f7c10003a00 2023-10-14T11:58:35.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.524+0000 7f7c277fe700 1 --2- 172.21.15.78:0/1269339829 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c10003a00 0x7f7c281e8390 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c000c00 tx=0x7f7c28152640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:35.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.525+0000 7f7c0ffff700 1 -- 172.21.15.78:0/1269339829 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c1c001670 con 0x7f7c10003a00 2023-10-14T11:58:35.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.525+0000 7f7c0ffff700 1 -- 172.21.15.78:0/1269339829 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c1c0070d0 con 0x7f7c10003a00 2023-10-14T11:58:35.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.525+0000 7f7c2df0d700 1 -- 172.21.15.78:0/1269339829 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c2814b820 con 0x7f7c10003a00 2023-10-14T11:58:35.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.525+0000 7f7c0ffff700 1 -- 172.21.15.78:0/1269339829 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c1c0074a0 con 0x7f7c10003a00 2023-10-14T11:58:35.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.525+0000 7f7c2df0d700 1 -- 172.21.15.78:0/1269339829 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c280c69c0 con 0x7f7c10003a00 2023-10-14T11:58:35.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.526+0000 7f7c2df0d700 1 -- 172.21.15.78:0/1269339829 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c281eb400 con 0x7f7c10003a00 2023-10-14T11:58:35.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.526+0000 7f7c0ffff700 1 -- 172.21.15.78:0/1269339829 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7c1c007600 con 0x7f7c10003a00 2023-10-14T11:58:35.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.526+0000 7f7c0ffff700 1 --2- 172.21.15.78:0/1269339829 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7c1406af00 0x7f7c1406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:35.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.527+0000 7f7c0ffff700 1 -- 172.21.15.78:0/1269339829 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7c1c08a4e0 con 0x7f7c10003a00 2023-10-14T11:58:35.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.527+0000 7f7c26ffd700 1 --2- 172.21.15.78:0/1269339829 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7c1406af00 0x7f7c1406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:35.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.527+0000 7f7c26ffd700 1 --2- 172.21.15.78:0/1269339829 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7c1406af00 0x7f7c1406d3c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7c2814c540 tx=0x7f7c18008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:35.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.531+0000 7f7c0ffff700 1 -- 172.21.15.78:0/1269339829 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7c1c055620 con 0x7f7c10003a00 2023-10-14T11:58:35.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.721+0000 7f7c2df0d700 1 -- 172.21.15.78:0/1269339829 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7c28074b80 con 0x7f7c10003a00 2023-10-14T11:58:35.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.721+0000 7f7c0ffff700 1 -- 172.21.15.78:0/1269339829 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7c1c05f070 con 0x7f7c10003a00 2023-10-14T11:58:35.723 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-10-14T11:58:35.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.723+0000 7f7c2df0d700 1 -- 172.21.15.78:0/1269339829 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7c1406af00 msgr2=0x7f7c1406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:35.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.723+0000 7f7c2df0d700 1 --2- 172.21.15.78:0/1269339829 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7c1406af00 0x7f7c1406d3c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7c2814c540 tx=0x7f7c18008040 comp rx=0 tx=0).stop 2023-10-14T11:58:35.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.723+0000 7f7c2df0d700 1 -- 172.21.15.78:0/1269339829 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c10003a00 msgr2=0x7f7c281e8390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:35.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.723+0000 7f7c2df0d700 1 --2- 172.21.15.78:0/1269339829 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c10003a00 0x7f7c281e8390 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c000c00 tx=0x7f7c28152640 comp rx=0 tx=0).stop 2023-10-14T11:58:35.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.723+0000 7f7c2df0d700 1 -- 172.21.15.78:0/1269339829 shutdown_connections 2023-10-14T11:58:35.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.723+0000 7f7c2df0d700 1 --2- 172.21.15.78:0/1269339829 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7c2814f4a0 0x7f7c281eb5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:35.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.723+0000 7f7c2df0d700 1 --2- 172.21.15.78:0/1269339829 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c10003a00 0x7f7c281e8390 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:35.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.723+0000 7f7c2df0d700 1 --2- 172.21.15.78:0/1269339829 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7c1406af00 0x7f7c1406d3c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:35.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.723+0000 7f7c2df0d700 1 --2- 172.21.15.78:0/1269339829 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7c2814c160 0x7f7c280ab850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:35.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.723+0000 7f7c2df0d700 1 -- 172.21.15.78:0/1269339829 >> 172.21.15.78:0/1269339829 conn(0x7f7c280c3a50 msgr2=0x7f7c28072160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:35.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.723+0000 7f7c2df0d700 1 -- 172.21.15.78:0/1269339829 shutdown_connections 2023-10-14T11:58:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.723+0000 7f7c2df0d700 1 -- 172.21.15.78:0/1269339829 wait complete. 2023-10-14T11:58:35.737 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T11:58:35.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.857+0000 7f65ec425700 1 Processor -- start 2023-10-14T11:58:35.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.857+0000 7f65ec425700 1 -- start start 2023-10-14T11:58:35.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.858+0000 7f65ec425700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f65e40c4540 0x7f65e40c02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:35.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.858+0000 7f65ec425700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f65e406b500 0x7f65e40c0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:35.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.858+0000 7f65ec425700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e406ebe0 con 0x7f65e413dbe0 2023-10-14T11:58:35.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.858+0000 7f65ec425700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65e406d330 con 0x7f65e406b500 2023-10-14T11:58:35.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.858+0000 7f65ec425700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e40c0d60 con 0x7f65e40c4540 2023-10-14T11:58:35.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.858+0000 7f65ea9c2700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f65e406b500 0x7f65e40c0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:35.866 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.858+0000 7f65ea9c2700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f65e406b500 0x7f65e40c0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45314/0 (socket says 172.21.15.78:45314) 2023-10-14T11:58:35.866 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.858+0000 7f65ea9c2700 1 -- 172.21.15.78:0/2691820735 learned_addr learned my addr 172.21.15.78:0/2691820735 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:35.866 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.858+0000 7f65ea1c1700 1 --1- 172.21.15.78:0/2691820735 >> v1:172.21.15.78:6789/0 conn(0x7f65e413dbe0 0x7f65e406a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50186/0 (socket says 172.21.15.78:50186) 2023-10-14T11:58:35.866 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.858+0000 7f65e99c0700 1 --2- 172.21.15.78:0/2691820735 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f65e40c4540 0x7f65e40c02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:35.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.859+0000 7f65ea9c2700 1 -- 172.21.15.78:0/2691820735 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f65e40c4540 msgr2=0x7f65e40c02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:35.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.859+0000 7f65ea9c2700 1 --2- 172.21.15.78:0/2691820735 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f65e40c4540 0x7f65e40c02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:35.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.859+0000 7f65ea9c2700 1 -- 172.21.15.78:0/2691820735 >> v1:172.21.15.78:6789/0 conn(0x7f65e413dbe0 legacy=0x7f65e406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:35.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.859+0000 7f65ea9c2700 1 -- 172.21.15.78:0/2691820735 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e41558c0 con 0x7f65e406b500 2023-10-14T11:58:35.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.859+0000 7f65e91bf700 1 -- 172.21.15.78:0/2691820735 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 518977816 0 0) 0x7f65e406ebe0 con 0x7f65e413dbe0 2023-10-14T11:58:35.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.859+0000 7f65e99c0700 1 --2- 172.21.15.78:0/2691820735 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f65e40c4540 0x7f65e40c02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:35.868 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.859+0000 7f65ea9c2700 1 --2- 172.21.15.78:0/2691820735 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65e406b500 0x7f65e40c0820 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f65dc009e50 tx=0x7f65dc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3c6fb8f6a423da4 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:35.868 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.859+0000 7f65e91bf700 1 -- 172.21.15.78:0/2691820735 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65dc00e070 con 0x7f65e406b500 2023-10-14T11:58:35.868 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.859+0000 7f65e91bf700 1 -- 172.21.15.78:0/2691820735 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65dc00b9c0 con 0x7f65e406b500 2023-10-14T11:58:35.868 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.859+0000 7f65e91bf700 1 -- 172.21.15.78:0/2691820735 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65dc00bd70 con 0x7f65e406b500 2023-10-14T11:58:35.868 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.859+0000 7f65ec425700 1 -- 172.21.15.78:0/2691820735 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65e406b500 msgr2=0x7f65e40c0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:35.869 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.859+0000 7f65ec425700 1 --2- 172.21.15.78:0/2691820735 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65e406b500 0x7f65e40c0820 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f65dc009e50 tx=0x7f65dc00b330 comp rx=0 tx=0).stop 2023-10-14T11:58:35.869 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.859+0000 7f65ec425700 1 -- 172.21.15.78:0/2691820735 shutdown_connections 2023-10-14T11:58:35.869 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.859+0000 7f65ec425700 1 --2- 172.21.15.78:0/2691820735 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65e406b500 0x7f65e40c0820 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:35.869 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.859+0000 7f65ec425700 1 --2- 172.21.15.78:0/2691820735 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f65e40c4540 0x7f65e40c02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:35.869 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.859+0000 7f65ec425700 1 -- 172.21.15.78:0/2691820735 >> 172.21.15.78:0/2691820735 conn(0x7f65e40c3a50 msgr2=0x7f65e40c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:35.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.859+0000 7f65ec425700 1 -- 172.21.15.78:0/2691820735 shutdown_connections 2023-10-14T11:58:35.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.859+0000 7f65ec425700 1 -- 172.21.15.78:0/2691820735 wait complete. 2023-10-14T11:58:35.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.861+0000 7f65ec425700 1 Processor -- start 2023-10-14T11:58:35.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.861+0000 7f65ec425700 1 -- start start 2023-10-14T11:58:35.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.861+0000 7f65ec425700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f65e406b500 0x7f65e40713f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:35.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.861+0000 7f65ec425700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65e40c4540 0x7f65e40ab280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:35.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.861+0000 7f65ec425700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f65e413dbe0 0x7f65e40ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:35.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.861+0000 7f65ec425700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e40ac4b0 con 0x7f65e40c4540 2023-10-14T11:58:35.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.861+0000 7f65ec425700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e41384c0 con 0x7f65e406b500 2023-10-14T11:58:35.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.861+0000 7f65ec425700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e4155e90 con 0x7f65e413dbe0 2023-10-14T11:58:35.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.861+0000 7f65e99c0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65e40c4540 0x7f65e40ab280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:35.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.861+0000 7f65e99c0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65e40c4540 0x7f65e40ab280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45318/0 (socket says 172.21.15.78:45318) 2023-10-14T11:58:35.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.861+0000 7f65e99c0700 1 -- 172.21.15.78:0/3160470802 learned_addr learned my addr 172.21.15.78:0/3160470802 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:35.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.861+0000 7f65ea1c1700 1 --2- 172.21.15.78:0/3160470802 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f65e406b500 0x7f65e40713f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:35.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.862+0000 7f65ea9c2700 1 --2- 172.21.15.78:0/3160470802 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f65e413dbe0 0x7f65e40ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.862+0000 7f65e99c0700 1 -- 172.21.15.78:0/3160470802 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f65e413dbe0 msgr2=0x7f65e40ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.862+0000 7f65e99c0700 1 --2- 172.21.15.78:0/3160470802 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f65e413dbe0 0x7f65e40ab7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.862+0000 7f65e99c0700 1 -- 172.21.15.78:0/3160470802 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f65e406b500 msgr2=0x7f65e40713f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.862+0000 7f65e99c0700 1 --2- 172.21.15.78:0/3160470802 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f65e406b500 0x7f65e40713f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.862+0000 7f65e99c0700 1 -- 172.21.15.78:0/3160470802 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65dc009b30 con 0x7f65e40c4540 2023-10-14T11:58:35.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.862+0000 7f65ea9c2700 1 --2- 172.21.15.78:0/3160470802 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f65e413dbe0 0x7f65e40ab7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:35.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.862+0000 7f65e99c0700 1 --2- 172.21.15.78:0/3160470802 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65e40c4540 0x7f65e40ab280 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f65e000e7a0 tx=0x7f65e000eab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:35.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.862+0000 7f65daffd700 1 -- 172.21.15.78:0/3160470802 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e0013610 con 0x7f65e40c4540 2023-10-14T11:58:35.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.862+0000 7f65daffd700 1 -- 172.21.15.78:0/3160470802 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65e0013770 con 0x7f65e40c4540 2023-10-14T11:58:35.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.862+0000 7f65ec425700 1 -- 172.21.15.78:0/3160470802 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65e40abf80 con 0x7f65e40c4540 2023-10-14T11:58:35.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.862+0000 7f65daffd700 1 -- 172.21.15.78:0/3160470802 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e0013ae0 con 0x7f65e40c4540 2023-10-14T11:58:35.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.862+0000 7f65ec425700 1 -- 172.21.15.78:0/3160470802 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65e41ea6e0 con 0x7f65e40c4540 2023-10-14T11:58:35.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.863+0000 7f65ec425700 1 -- 172.21.15.78:0/3160470802 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65e41ea540 con 0x7f65e40c4540 2023-10-14T11:58:35.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.863+0000 7f65daffd700 1 -- 172.21.15.78:0/3160470802 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f65e0013c40 con 0x7f65e40c4540 2023-10-14T11:58:35.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.864+0000 7f65daffd700 1 --2- 172.21.15.78:0/3160470802 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f65c806af00 0x7f65c806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:35.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.864+0000 7f65daffd700 1 -- 172.21.15.78:0/3160470802 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f65e00890e0 con 0x7f65e40c4540 2023-10-14T11:58:35.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.864+0000 7f65ea1c1700 1 --2- 172.21.15.78:0/3160470802 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f65c806af00 0x7f65c806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:35.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.864+0000 7f65ea1c1700 1 --2- 172.21.15.78:0/3160470802 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f65c806af00 0x7f65c806d3c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f65d4000f60 tx=0x7f65d4007e70 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:35.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:35.868+0000 7f65daffd700 1 -- 172.21.15.78:0/3160470802 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f65e41ea540 con 0x7f65e40c4540 2023-10-14T11:58:36.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.058+0000 7f65ec425700 1 -- 172.21.15.78:0/3160470802 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f65e41ea540 con 0x7f65e40c4540 2023-10-14T11:58:36.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.058+0000 7f65daffd700 1 -- 172.21.15.78:0/3160470802 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f65e41ea540 con 0x7f65e40c4540 2023-10-14T11:58:36.060 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2023-10-14T11:58:36.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.060+0000 7f65ec425700 1 -- 172.21.15.78:0/3160470802 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f65c806af00 msgr2=0x7f65c806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:36.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.060+0000 7f65ec425700 1 --2- 172.21.15.78:0/3160470802 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f65c806af00 0x7f65c806d3c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f65d4000f60 tx=0x7f65d4007e70 comp rx=0 tx=0).stop 2023-10-14T11:58:36.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.060+0000 7f65ec425700 1 -- 172.21.15.78:0/3160470802 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65e40c4540 msgr2=0x7f65e40ab280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:36.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.060+0000 7f65ec425700 1 --2- 172.21.15.78:0/3160470802 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65e40c4540 0x7f65e40ab280 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f65e000e7a0 tx=0x7f65e000eab0 comp rx=0 tx=0).stop 2023-10-14T11:58:36.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.060+0000 7f65ec425700 1 -- 172.21.15.78:0/3160470802 shutdown_connections 2023-10-14T11:58:36.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.060+0000 7f65ec425700 1 --2- 172.21.15.78:0/3160470802 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f65e413dbe0 0x7f65e40ab7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:36.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.060+0000 7f65ec425700 1 --2- 172.21.15.78:0/3160470802 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f65c806af00 0x7f65c806d3c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:36.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.060+0000 7f65ec425700 1 --2- 172.21.15.78:0/3160470802 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f65e406b500 0x7f65e40713f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:36.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.060+0000 7f65ec425700 1 --2- 172.21.15.78:0/3160470802 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65e40c4540 0x7f65e40ab280 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:36.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.060+0000 7f65ec425700 1 -- 172.21.15.78:0/3160470802 >> 172.21.15.78:0/3160470802 conn(0x7f65e40c3a50 msgr2=0x7f65e413c580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:36.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.060+0000 7f65ec425700 1 -- 172.21.15.78:0/3160470802 shutdown_connections 2023-10-14T11:58:36.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.060+0000 7f65ec425700 1 -- 172.21.15.78:0/3160470802 wait complete. 2023-10-14T11:58:36.074 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T11:58:36.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.192+0000 7f983156b700 1 Processor -- start 2023-10-14T11:58:36.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.192+0000 7f983156b700 1 -- start start 2023-10-14T11:58:36.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.193+0000 7f983156b700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f982c06ab40 0x7f982c06f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:36.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.193+0000 7f983156b700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f982c1554e0 0x7f982c06af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:36.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.193+0000 7f983156b700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f982c0c49f0 con 0x7f982c0c0350 2023-10-14T11:58:36.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.193+0000 7f983156b700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f982c0726a0 con 0x7f982c06ab40 2023-10-14T11:58:36.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.193+0000 7f983156b700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f982c072980 con 0x7f982c1554e0 2023-10-14T11:58:36.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.193+0000 7f982affd700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f982c06ab40 0x7f982c06f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:36.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.193+0000 7f982b7fe700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f982c0c0350 0x7f982c13d7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50204/0 (socket says 172.21.15.78:50204) 2023-10-14T11:58:36.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.193+0000 7f982b7fe700 1 -- 172.21.15.78:0/1764013146 learned_addr learned my addr 172.21.15.78:0/1764013146 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:36.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.193+0000 7f982a7fc700 1 --2- 172.21.15.78:0/1764013146 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f982c1554e0 0x7f982c06af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:36.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.193+0000 7f982affd700 1 -- 172.21.15.78:0/1764013146 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f982c1554e0 msgr2=0x7f982c06af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:36.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.193+0000 7f982affd700 1 --2- 172.21.15.78:0/1764013146 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f982c1554e0 0x7f982c06af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:36.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.193+0000 7f982affd700 1 -- 172.21.15.78:0/1764013146 >> v1:172.21.15.78:6789/0 conn(0x7f982c0c0350 legacy=0x7f982c13d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:36.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.193+0000 7f982affd700 1 -- 172.21.15.78:0/1764013146 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f982c154a00 con 0x7f982c06ab40 2023-10-14T11:58:36.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.193+0000 7f9829ffb700 1 -- 172.21.15.78:0/1764013146 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3406784843 0 0) 0x7f982c0c49f0 con 0x7f982c0c0350 2023-10-14T11:58:36.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.193+0000 7f982affd700 1 --2- 172.21.15.78:0/1764013146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f982c06ab40 0x7f982c06f1e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9820009840 tx=0x7f9820009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4938745e0ddf0ee0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:36.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.193+0000 7f9829ffb700 1 -- 172.21.15.78:0/1764013146 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9820011830 con 0x7f982c06ab40 2023-10-14T11:58:36.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.193+0000 7f9829ffb700 1 -- 172.21.15.78:0/1764013146 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9820011990 con 0x7f982c06ab40 2023-10-14T11:58:36.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.194+0000 7f9829ffb700 1 -- 172.21.15.78:0/1764013146 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9820011d20 con 0x7f982c06ab40 2023-10-14T11:58:36.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.194+0000 7f983156b700 1 -- 172.21.15.78:0/1764013146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f982c06ab40 msgr2=0x7f982c06f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:36.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.194+0000 7f983156b700 1 --2- 172.21.15.78:0/1764013146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f982c06ab40 0x7f982c06f1e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9820009840 tx=0x7f9820009b50 comp rx=0 tx=0).stop 2023-10-14T11:58:36.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.194+0000 7f983156b700 1 -- 172.21.15.78:0/1764013146 shutdown_connections 2023-10-14T11:58:36.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.194+0000 7f983156b700 1 --2- 172.21.15.78:0/1764013146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f982c06ab40 0x7f982c06f1e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:36.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.194+0000 7f983156b700 1 --2- 172.21.15.78:0/1764013146 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f982c1554e0 0x7f982c06af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:36.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.194+0000 7f983156b700 1 -- 172.21.15.78:0/1764013146 >> 172.21.15.78:0/1764013146 conn(0x7f982c0c3ac0 msgr2=0x7f982c0c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:36.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.194+0000 7f983156b700 1 -- 172.21.15.78:0/1764013146 shutdown_connections 2023-10-14T11:58:36.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.194+0000 7f983156b700 1 -- 172.21.15.78:0/1764013146 wait complete. 2023-10-14T11:58:36.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.196+0000 7f983156b700 1 Processor -- start 2023-10-14T11:58:36.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.196+0000 7f983156b700 1 -- start start 2023-10-14T11:58:36.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.197+0000 7f983156b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f982c06ab40 0x7f982c137c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:36.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.197+0000 7f983156b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f982c0c0350 0x7f982c138190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:36.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.197+0000 7f983156b700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f982c1554e0 0x7f982c1ed240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:36.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.197+0000 7f983156b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982c158990 con 0x7f982c0c0350 2023-10-14T11:58:36.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.197+0000 7f983156b700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982c1516a0 con 0x7f982c06ab40 2023-10-14T11:58:36.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.197+0000 7f983156b700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f982c151980 con 0x7f982c1554e0 2023-10-14T11:58:36.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.197+0000 7f982a7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f982c0c0350 0x7f982c138190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:36.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.197+0000 7f982a7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f982c0c0350 0x7f982c138190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45332/0 (socket says 172.21.15.78:45332) 2023-10-14T11:58:36.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.197+0000 7f982a7fc700 1 -- 172.21.15.78:0/3686660353 learned_addr learned my addr 172.21.15.78:0/3686660353 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:36.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.197+0000 7f982b7fe700 1 --2- 172.21.15.78:0/3686660353 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f982c1554e0 0x7f982c1ed240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:36.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.197+0000 7f982affd700 1 --2- 172.21.15.78:0/3686660353 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f982c06ab40 0x7f982c137c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:36.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.197+0000 7f982a7fc700 1 -- 172.21.15.78:0/3686660353 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f982c1554e0 msgr2=0x7f982c1ed240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:36.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.197+0000 7f982a7fc700 1 --2- 172.21.15.78:0/3686660353 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f982c1554e0 0x7f982c1ed240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:36.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.197+0000 7f982a7fc700 1 -- 172.21.15.78:0/3686660353 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f982c06ab40 msgr2=0x7f982c137c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:36.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.197+0000 7f982a7fc700 1 --2- 172.21.15.78:0/3686660353 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f982c06ab40 0x7f982c137c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:36.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.197+0000 7f982a7fc700 1 -- 172.21.15.78:0/3686660353 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98200094a0 con 0x7f982c0c0350 2023-10-14T11:58:36.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.197+0000 7f982b7fe700 1 --2- 172.21.15.78:0/3686660353 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f982c1554e0 0x7f982c1ed240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:36.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.197+0000 7f982a7fc700 1 --2- 172.21.15.78:0/3686660353 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f982c0c0350 0x7f982c138190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f981c00d800 tx=0x7f981c00db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:36.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.197+0000 7f98137fe700 1 -- 172.21.15.78:0/3686660353 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981c0125f0 con 0x7f982c0c0350 2023-10-14T11:58:36.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.197+0000 7f98137fe700 1 -- 172.21.15.78:0/3686660353 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f981c012750 con 0x7f982c0c0350 2023-10-14T11:58:36.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.197+0000 7f983156b700 1 -- 172.21.15.78:0/3686660353 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f982c1eaed0 con 0x7f982c0c0350 2023-10-14T11:58:36.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.197+0000 7f98137fe700 1 -- 172.21.15.78:0/3686660353 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981c012b00 con 0x7f982c0c0350 2023-10-14T11:58:36.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.197+0000 7f983156b700 1 -- 172.21.15.78:0/3686660353 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f982c1eb490 con 0x7f982c0c0350 2023-10-14T11:58:36.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.198+0000 7f983156b700 1 -- 172.21.15.78:0/3686660353 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f982c1eb680 con 0x7f982c0c0350 2023-10-14T11:58:36.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.199+0000 7f98137fe700 1 -- 172.21.15.78:0/3686660353 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f981c012c60 con 0x7f982c0c0350 2023-10-14T11:58:36.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.199+0000 7f98137fe700 1 --2- 172.21.15.78:0/3686660353 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f981806af00 0x7f981806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:36.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.199+0000 7f98137fe700 1 -- 172.21.15.78:0/3686660353 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f981c0898d0 con 0x7f982c0c0350 2023-10-14T11:58:36.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.199+0000 7f982affd700 1 --2- 172.21.15.78:0/3686660353 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f981806af00 0x7f981806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:36.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.199+0000 7f982affd700 1 --2- 172.21.15.78:0/3686660353 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f981806af00 0x7f981806d3c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f98200090b0 tx=0x7f9820008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:36.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.203+0000 7f98137fe700 1 -- 172.21.15.78:0/3686660353 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f981c054a10 con 0x7f982c0c0350 2023-10-14T11:58:36.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.395+0000 7f983156b700 1 -- 172.21.15.78:0/3686660353 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f982c074b80 con 0x7f982c0c0350 2023-10-14T11:58:36.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.396+0000 7f98137fe700 1 -- 172.21.15.78:0/3686660353 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16481 (secure 0 0 0) 0x7f981c058030 con 0x7f982c0c0350 2023-10-14T11:58:36.398 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:36.398 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":21,"fsid":"4c5dc134-3092-406a-a1e1-13e6d9e926e8","created":"2023-10-14T11:57:57.363750+0000","modified":"2023-10-14T11:58:16.139413+0000","last_up_change":"2023-10-14T11:58:06.982472+0000","last_in_change":"2023-10-14T11:57:59.821373+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T11:58:09.001200+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T11:58:12.778573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T11:58:13.490286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"625a5fc0-9474-4eae-b80e-14c27a74ad19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6801","nonce":3884969780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6803","nonce":3884969780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6807","nonce":3884969780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6805","nonce":3884969780}]},"public_addr":"172.21.15.78:6801/3884969780","cluster_addr":"172.21.15.78:6803/3884969780","heartbeat_back_addr":"172.21.15.78:6807/3884969780","heartbeat_front_addr":"172.21.15.78:6805/3884969780","state":["exists","up"]},{"osd":1,"uuid":"49d63db1-2981-49ab-9c40-c50fc744b982","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6809","nonce":2176928406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6811","nonce":2176928406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6815","nonce":2176928406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6813","nonce":2176928406}]},"public_addr":"172.21.15.78:6809/2176928406","cluster_addr":"172.21.15.78:6811/2176928406","heartbeat_back_addr":"172.21.15.78:6815/2176928406","heartbeat_front_addr":"172.21.15.78:6813/2176928406","state":["exists","up"]},{"osd":2,"uuid":"1d7d0b29-cbd1-4717-9fe6-765a8d4998b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6825","nonce":3647044577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6827","nonce":3647044577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6831","nonce":3647044577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6829","nonce":3647044577}]},"public_addr":"172.21.15.78:6825/3647044577","cluster_addr":"172.21.15.78:6827/3647044577","heartbeat_back_addr":"172.21.15.78:6831/3647044577","heartbeat_front_addr":"172.21.15.78:6829/3647044577","state":["exists","up"]},{"osd":3,"uuid":"2f75ded1-e9e2-40e7-99ae-45e1e425e1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6817","nonce":3584112288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6819","nonce":3584112288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6823","nonce":3584112288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6821","nonce":3584112288}]},"public_addr":"172.21.15.78:6817/3584112288","cluster_addr":"172.21.15.78:6819/3584112288","heartbeat_back_addr":"172.21.15.78:6823/3584112288","heartbeat_front_addr":"172.21.15.78:6821/3584112288","state":["exists","up"]},{"osd":4,"uuid":"5130b432-8358-46cb-8a31-3064c976f6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6801","nonce":3732756301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6803","nonce":3732756301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6807","nonce":3732756301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6805","nonce":3732756301}]},"public_addr":"172.21.15.195:6801/3732756301","cluster_addr":"172.21.15.195:6803/3732756301","heartbeat_back_addr":"172.21.15.195:6807/3732756301","heartbeat_front_addr":"172.21.15.195:6805/3732756301","state":["exists","up"]},{"osd":5,"uuid":"60e234a9-a506-4e40-9c19-49a58f3cf28c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6825","nonce":2857706411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6827","nonce":2857706411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6831","nonce":2857706411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6829","nonce":2857706411}]},"public_addr":"172.21.15.195:6825/2857706411","cluster_addr":"172.21.15.195:6827/2857706411","heartbeat_back_addr":"172.21.15.195:6831/2857706411","heartbeat_front_addr":"172.21.15.195:6829/2857706411","state":["exists","up"]},{"osd":6,"uuid":"df7d2dd6-26b6-4eb1-96e5-94d1f3955b33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6810","nonce":72554374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6813","nonce":72554374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6820","nonce":72554374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6817","nonce":72554374}]},"public_addr":"172.21.15.195:6810/72554374","cluster_addr":"172.21.15.195:6813/72554374","heartbeat_back_addr":"172.21.15.195:6820/72554374","heartbeat_front_addr":"172.21.15.195:6817/72554374","state":["exists","up"]},{"osd":7,"uuid":"47a9d4de-f56d-4aa4-9900-c4a9cf9b2f97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6809","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6811","nonce":482472979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6816","nonce":482472979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6823","nonce":482472979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6821","nonce":482472979}]},"public_addr":"172.21.15.195:6811/482472979","cluster_addr":"172.21.15.195:6816/482472979","heartbeat_back_addr":"172.21.15.195:6823/482472979","heartbeat_front_addr":"172.21.15.195:6821/482472979","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.754003+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.005994+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.043603+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.024506+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.062585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.033767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.994825+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.038300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T11:58:36.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.398+0000 7f983156b700 1 -- 172.21.15.78:0/3686660353 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f981806af00 msgr2=0x7f981806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:36.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.398+0000 7f983156b700 1 --2- 172.21.15.78:0/3686660353 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f981806af00 0x7f981806d3c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f98200090b0 tx=0x7f9820008030 comp rx=0 tx=0).stop 2023-10-14T11:58:36.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.398+0000 7f983156b700 1 -- 172.21.15.78:0/3686660353 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f982c0c0350 msgr2=0x7f982c138190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:36.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.398+0000 7f983156b700 1 --2- 172.21.15.78:0/3686660353 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f982c0c0350 0x7f982c138190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f981c00d800 tx=0x7f981c00db10 comp rx=0 tx=0).stop 2023-10-14T11:58:36.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.398+0000 7f983156b700 1 -- 172.21.15.78:0/3686660353 shutdown_connections 2023-10-14T11:58:36.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.398+0000 7f983156b700 1 --2- 172.21.15.78:0/3686660353 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f982c1554e0 0x7f982c1ed240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:36.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.398+0000 7f983156b700 1 --2- 172.21.15.78:0/3686660353 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f981806af00 0x7f981806d3c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:36.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.398+0000 7f983156b700 1 --2- 172.21.15.78:0/3686660353 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f982c06ab40 0x7f982c137c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:36.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.398+0000 7f983156b700 1 --2- 172.21.15.78:0/3686660353 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f982c0c0350 0x7f982c138190 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:36.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.398+0000 7f983156b700 1 -- 172.21.15.78:0/3686660353 >> 172.21.15.78:0/3686660353 conn(0x7f982c0c3ac0 msgr2=0x7f982c152b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:36.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.398+0000 7f983156b700 1 -- 172.21.15.78:0/3686660353 shutdown_connections 2023-10-14T11:58:36.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.398+0000 7f983156b700 1 -- 172.21.15.78:0/3686660353 wait complete. 2023-10-14T11:58:36.412 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T11:58:09.001200+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T11:58:12.778573+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T11:58:13.490286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T11:58:36.413 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T11:58:36.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.533+0000 7fc72783b700 1 Processor -- start 2023-10-14T11:58:36.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.533+0000 7fc72783b700 1 -- start start 2023-10-14T11:58:36.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.534+0000 7fc72783b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fc72013dbe0 0x7fc72006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:36.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.534+0000 7fc72783b700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc7200c4540 0x7fc7200c02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:36.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.534+0000 7fc72783b700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc72006ebe0 con 0x7fc72006ae10 2023-10-14T11:58:36.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.534+0000 7fc72783b700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc72006d330 con 0x7fc72013dbe0 2023-10-14T11:58:36.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.534+0000 7fc72783b700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7201558c0 con 0x7fc7200c4540 2023-10-14T11:58:36.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.534+0000 7fc725dd8700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc72006ae10 0x7fc7200c0820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50218/0 (socket says 172.21.15.78:50218) 2023-10-14T11:58:36.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.534+0000 7fc725dd8700 1 -- 172.21.15.78:0/3398272202 learned_addr learned my addr 172.21.15.78:0/3398272202 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:36.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.534+0000 7fc724dd6700 1 --2- 172.21.15.78:0/3398272202 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc7200c4540 0x7fc7200c02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:36.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.534+0000 7fc717fff700 1 -- 172.21.15.78:0/3398272202 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3962903380 0 0) 0x7fc72006ebe0 con 0x7fc72006ae10 2023-10-14T11:58:36.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.534+0000 7fc7255d7700 1 --2- 172.21.15.78:0/3398272202 >> v2:172.21.15.195:3300/0 conn(0x7fc72013dbe0 0x7fc72006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:36.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.534+0000 7fc717fff700 1 -- 172.21.15.78:0/3398272202 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc70c0035b0 con 0x7fc72006ae10 2023-10-14T11:58:36.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.534+0000 7fc717fff700 1 -- 172.21.15.78:0/3398272202 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1323447680 0 0) 0x7fc70c0035b0 con 0x7fc72006ae10 2023-10-14T11:58:36.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.535+0000 7fc717fff700 1 -- 172.21.15.78:0/3398272202 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc7200c4540 msgr2=0x7fc7200c02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:36.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.535+0000 7fc717fff700 1 --2- 172.21.15.78:0/3398272202 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc7200c4540 0x7fc7200c02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:36.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.535+0000 7fc717fff700 1 -- 172.21.15.78:0/3398272202 >> v2:172.21.15.195:3300/0 conn(0x7fc72013dbe0 msgr2=0x7fc72006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:36.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.535+0000 7fc717fff700 1 --2- 172.21.15.78:0/3398272202 >> v2:172.21.15.195:3300/0 conn(0x7fc72013dbe0 0x7fc72006a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:36.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.535+0000 7fc717fff700 1 -- 172.21.15.78:0/3398272202 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc720154b00 con 0x7fc72006ae10 2023-10-14T11:58:36.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.535+0000 7fc717fff700 1 -- 172.21.15.78:0/3398272202 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441799917 0 0) 0x7fc7180019f0 con 0x7fc72006ae10 2023-10-14T11:58:36.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.535+0000 7fc717fff700 1 -- 172.21.15.78:0/3398272202 >> v1:172.21.15.78:6789/0 conn(0x7fc72006ae10 legacy=0x7fc7200c0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:36.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.535+0000 7fc717fff700 1 --2- 172.21.15.78:0/3398272202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc70c003a00 0x7fc70c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:36.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.535+0000 7fc717fff700 1 --2- 172.21.15.78:0/3398272202 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc70c004850 0x7fc70c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:36.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.535+0000 7fc717fff700 1 --2- 172.21.15.78:0/3398272202 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc70c005210 0x7fc70c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:36.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.535+0000 7fc717fff700 1 -- 172.21.15.78:0/3398272202 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72006d330 con 0x7fc70c003a00 2023-10-14T11:58:36.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.535+0000 7fc717fff700 1 -- 172.21.15.78:0/3398272202 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7201558c0 con 0x7fc70c005210 2023-10-14T11:58:36.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.535+0000 7fc717fff700 1 -- 172.21.15.78:0/3398272202 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc70c00c050 con 0x7fc70c004850 2023-10-14T11:58:36.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.535+0000 7fc724dd6700 1 --2- 172.21.15.78:0/3398272202 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc7200c4540 0x7fc7200c02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:36.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.535+0000 7fc7255d7700 1 --2- 172.21.15.78:0/3398272202 >> v2:172.21.15.195:3300/0 conn(0x7fc72013dbe0 0x7fc72006a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:36.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.535+0000 7fc7255d7700 1 --2- 172.21.15.78:0/3398272202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc70c003a00 0x7fc70c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:36.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.535+0000 7fc725dd8700 1 --2- 172.21.15.78:0/3398272202 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc70c005210 0x7fc70c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:36.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.535+0000 7fc724dd6700 1 --2- 172.21.15.78:0/3398272202 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc70c004850 0x7fc70c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:36.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.535+0000 7fc7255d7700 1 -- 172.21.15.78:0/3398272202 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc70c004850 msgr2=0x7fc70c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:36.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.535+0000 7fc7255d7700 1 --2- 172.21.15.78:0/3398272202 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc70c004850 0x7fc70c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:36.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.535+0000 7fc7255d7700 1 -- 172.21.15.78:0/3398272202 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc70c005210 msgr2=0x7fc70c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:36.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.535+0000 7fc7255d7700 1 --2- 172.21.15.78:0/3398272202 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc70c005210 0x7fc70c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:36.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.535+0000 7fc7255d7700 1 -- 172.21.15.78:0/3398272202 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc70c0035b0 con 0x7fc70c003a00 2023-10-14T11:58:36.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.535+0000 7fc725dd8700 1 --2- 172.21.15.78:0/3398272202 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc70c005210 0x7fc70c00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:36.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.535+0000 7fc724dd6700 1 --2- 172.21.15.78:0/3398272202 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc70c004850 0x7fc70c004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:36.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.535+0000 7fc7255d7700 1 --2- 172.21.15.78:0/3398272202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc70c003a00 0x7fc70c003e20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc710009960 tx=0x7fc70c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e797f5b334ab5e6f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:36.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.535+0000 7fc717fff700 1 -- 172.21.15.78:0/3398272202 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71000c070 con 0x7fc70c003a00 2023-10-14T11:58:36.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.535+0000 7fc717fff700 1 -- 172.21.15.78:0/3398272202 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc710004230 con 0x7fc70c003a00 2023-10-14T11:58:36.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.535+0000 7fc717fff700 1 -- 172.21.15.78:0/3398272202 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7100036a0 con 0x7fc70c003a00 2023-10-14T11:58:36.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.536+0000 7fc72783b700 1 -- 172.21.15.78:0/3398272202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc70c003a00 msgr2=0x7fc70c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:36.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.536+0000 7fc72783b700 1 --2- 172.21.15.78:0/3398272202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc70c003a00 0x7fc70c003e20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc710009960 tx=0x7fc70c005880 comp rx=0 tx=0).stop 2023-10-14T11:58:36.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.536+0000 7fc725dd8700 1 -- 172.21.15.78:0/3398272202 reap_dead start 2023-10-14T11:58:36.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.536+0000 7fc72783b700 1 -- 172.21.15.78:0/3398272202 shutdown_connections 2023-10-14T11:58:36.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.536+0000 7fc72783b700 1 -- 172.21.15.78:0/3398272202 >> 172.21.15.78:0/3398272202 conn(0x7fc7200c3a50 msgr2=0x7fc7200c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:36.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.536+0000 7fc72783b700 1 -- 172.21.15.78:0/3398272202 shutdown_connections 2023-10-14T11:58:36.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.536+0000 7fc72783b700 1 -- 172.21.15.78:0/3398272202 wait complete. 2023-10-14T11:58:36.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.537+0000 7fc72783b700 1 Processor -- start 2023-10-14T11:58:36.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.537+0000 7fc72783b700 1 -- start start 2023-10-14T11:58:36.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.538+0000 7fc72783b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc70c003a00 0x7fc7201ed8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:36.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.538+0000 7fc72783b700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc720156a10 0x7fc7201582d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:36.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.538+0000 7fc72783b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc720138140 0x7fc7201e9800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:36.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.538+0000 7fc72783b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7201ed440 con 0x7fc70c003a00 2023-10-14T11:58:36.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.538+0000 7fc72783b700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7201ece10 con 0x7fc720138140 2023-10-14T11:58:36.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.538+0000 7fc72783b700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc720158810 con 0x7fc720156a10 2023-10-14T11:58:36.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.538+0000 7fc7255d7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc70c003a00 0x7fc7201ed8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:36.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.538+0000 7fc7255d7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc70c003a00 0x7fc7201ed8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45350/0 (socket says 172.21.15.78:45350) 2023-10-14T11:58:36.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.538+0000 7fc7255d7700 1 -- 172.21.15.78:0/2263002083 learned_addr learned my addr 172.21.15.78:0/2263002083 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:36.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.538+0000 7fc7255d7700 1 -- 172.21.15.78:0/2263002083 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc720156a10 msgr2=0x7fc7201582d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:36.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.538+0000 7fc724dd6700 1 --2- 172.21.15.78:0/2263002083 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc720156a10 0x7fc7201582d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:36.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.538+0000 7fc7255d7700 1 --2- 172.21.15.78:0/2263002083 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc720156a10 0x7fc7201582d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:36.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.538+0000 7fc7255d7700 1 -- 172.21.15.78:0/2263002083 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc720138140 msgr2=0x7fc7201e9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:36.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.538+0000 7fc7255d7700 1 --2- 172.21.15.78:0/2263002083 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc720138140 0x7fc7201e9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:36.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.538+0000 7fc7255d7700 1 -- 172.21.15.78:0/2263002083 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc720158ad0 con 0x7fc70c003a00 2023-10-14T11:58:36.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.538+0000 7fc7255d7700 1 --2- 172.21.15.78:0/2263002083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc70c003a00 0x7fc7201ed8d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc70c0057f0 tx=0x7fc7100047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:36.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.539+0000 7fc715ffb700 1 -- 172.21.15.78:0/2263002083 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71001c460 con 0x7fc70c003a00 2023-10-14T11:58:36.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.539+0000 7fc715ffb700 1 -- 172.21.15.78:0/2263002083 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc71001c5e0 con 0x7fc70c003a00 2023-10-14T11:58:36.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.539+0000 7fc715ffb700 1 -- 172.21.15.78:0/2263002083 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71001c990 con 0x7fc70c003a00 2023-10-14T11:58:36.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.539+0000 7fc72783b700 1 -- 172.21.15.78:0/2263002083 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7201ed0d0 con 0x7fc70c003a00 2023-10-14T11:58:36.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.539+0000 7fc72783b700 1 -- 172.21.15.78:0/2263002083 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7201385c0 con 0x7fc70c003a00 2023-10-14T11:58:36.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.540+0000 7fc72783b700 1 -- 172.21.15.78:0/2263002083 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc720138750 con 0x7fc70c003a00 2023-10-14T11:58:36.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.540+0000 7fc715ffb700 1 -- 172.21.15.78:0/2263002083 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc71001caf0 con 0x7fc70c003a00 2023-10-14T11:58:36.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.540+0000 7fc715ffb700 1 --2- 172.21.15.78:0/2263002083 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc70406aed0 0x7fc70406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:36.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.541+0000 7fc715ffb700 1 -- 172.21.15.78:0/2263002083 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc71008a960 con 0x7fc70c003a00 2023-10-14T11:58:36.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.541+0000 7fc724dd6700 1 --2- 172.21.15.78:0/2263002083 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc70406aed0 0x7fc70406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:36.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.541+0000 7fc724dd6700 1 --2- 172.21.15.78:0/2263002083 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc70406aed0 0x7fc70406d390 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc7200c5f50 tx=0x7fc71c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:36.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.545+0000 7fc715ffb700 1 -- 172.21.15.78:0/2263002083 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc710055aa0 con 0x7fc70c003a00 2023-10-14T11:58:36.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.733+0000 7fc72783b700 1 -- 172.21.15.78:0/2263002083 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc7201ed720 con 0x7fc70c003a00 2023-10-14T11:58:36.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.734+0000 7fc715ffb700 1 -- 172.21.15.78:0/2263002083 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc7100590c0 con 0x7fc70c003a00 2023-10-14T11:58:36.736 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-10-14T11:58:36.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.736+0000 7fc72783b700 1 -- 172.21.15.78:0/2263002083 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc70406aed0 msgr2=0x7fc70406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:36.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.736+0000 7fc72783b700 1 --2- 172.21.15.78:0/2263002083 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc70406aed0 0x7fc70406d390 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc7200c5f50 tx=0x7fc71c008040 comp rx=0 tx=0).stop 2023-10-14T11:58:36.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.736+0000 7fc72783b700 1 -- 172.21.15.78:0/2263002083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc70c003a00 msgr2=0x7fc7201ed8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:36.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.736+0000 7fc72783b700 1 --2- 172.21.15.78:0/2263002083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc70c003a00 0x7fc7201ed8d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc70c0057f0 tx=0x7fc7100047c0 comp rx=0 tx=0).stop 2023-10-14T11:58:36.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.736+0000 7fc72783b700 1 -- 172.21.15.78:0/2263002083 shutdown_connections 2023-10-14T11:58:36.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.736+0000 7fc72783b700 1 --2- 172.21.15.78:0/2263002083 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc70406aed0 0x7fc70406d390 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:36.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.736+0000 7fc72783b700 1 --2- 172.21.15.78:0/2263002083 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc720138140 0x7fc7201e9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:36.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.736+0000 7fc72783b700 1 --2- 172.21.15.78:0/2263002083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc70c003a00 0x7fc7201ed8d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:36.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.736+0000 7fc72783b700 1 --2- 172.21.15.78:0/2263002083 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc720156a10 0x7fc7201582d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:36.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.736+0000 7fc72783b700 1 -- 172.21.15.78:0/2263002083 >> 172.21.15.78:0/2263002083 conn(0x7fc7200c3a50 msgr2=0x7fc7200c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:36.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.736+0000 7fc72783b700 1 -- 172.21.15.78:0/2263002083 shutdown_connections 2023-10-14T11:58:36.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.736+0000 7fc72783b700 1 -- 172.21.15.78:0/2263002083 wait complete. 2023-10-14T11:58:36.750 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T11:58:36.873 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.872+0000 7f267d43b700 1 Processor -- start 2023-10-14T11:58:36.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.872+0000 7f267d43b700 1 -- start start 2023-10-14T11:58:36.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.872+0000 7f267d43b700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f26780c6cd0 0x7f26780c70b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:36.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.872+0000 7f267d43b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f26780bd950 0x7f26781588e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:36.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.872+0000 7f267d43b700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26780c4c40 con 0x7f267814ebd0 2023-10-14T11:58:36.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.872+0000 7f267d43b700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2678158e20 con 0x7f26780c6cd0 2023-10-14T11:58:36.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.872+0000 7f267d43b700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26780c54e0 con 0x7f26780bd950 2023-10-14T11:58:36.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.872+0000 7f2676ffd700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f26780c6cd0 0x7f26780c70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:36.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.873+0000 7f2676ffd700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f26780c6cd0 0x7f26780c70b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45358/0 (socket says 172.21.15.78:45358) 2023-10-14T11:58:36.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.873+0000 7f2676ffd700 1 -- 172.21.15.78:0/2560276372 learned_addr learned my addr 172.21.15.78:0/2560276372 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:36.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.873+0000 7f26777fe700 1 --2- 172.21.15.78:0/2560276372 >> v2:172.21.15.195:3300/0 conn(0x7f26780bd950 0x7f26781588e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:36.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.873+0000 7f2676ffd700 1 -- 172.21.15.78:0/2560276372 >> v2:172.21.15.195:3300/0 conn(0x7f26780bd950 msgr2=0x7f26781588e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:36.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.873+0000 7f2676ffd700 1 --2- 172.21.15.78:0/2560276372 >> v2:172.21.15.195:3300/0 conn(0x7f26780bd950 0x7f26781588e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:36.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.873+0000 7f2676ffd700 1 -- 172.21.15.78:0/2560276372 >> v1:172.21.15.195:6789/0 conn(0x7f267814ebd0 legacy=0x7f26781581d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:36.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.873+0000 7f2676ffd700 1 -- 172.21.15.78:0/2560276372 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26781556d0 con 0x7f26780c6cd0 2023-10-14T11:58:36.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.873+0000 7f2676ffd700 1 --2- 172.21.15.78:0/2560276372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26780c6cd0 0x7f26780c70b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f267808c430 tx=0x7f2668009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6573a3b480345ffc server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:36.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.873+0000 7f2675ffb700 1 -- 172.21.15.78:0/2560276372 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2668004230 con 0x7f26780c6cd0 2023-10-14T11:58:36.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.873+0000 7f2675ffb700 1 -- 172.21.15.78:0/2560276372 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2668004390 con 0x7f26780c6cd0 2023-10-14T11:58:36.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.873+0000 7f2675ffb700 1 -- 172.21.15.78:0/2560276372 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26680036a0 con 0x7f26780c6cd0 2023-10-14T11:58:36.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.874+0000 7f267d43b700 1 -- 172.21.15.78:0/2560276372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26780c6cd0 msgr2=0x7f26780c70b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:36.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.874+0000 7f267d43b700 1 --2- 172.21.15.78:0/2560276372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26780c6cd0 0x7f26780c70b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f267808c430 tx=0x7f2668009bd0 comp rx=0 tx=0).stop 2023-10-14T11:58:36.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.874+0000 7f267d43b700 1 -- 172.21.15.78:0/2560276372 shutdown_connections 2023-10-14T11:58:36.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.874+0000 7f267d43b700 1 --2- 172.21.15.78:0/2560276372 >> v2:172.21.15.195:3300/0 conn(0x7f26780bd950 0x7f26781588e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:36.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.874+0000 7f267d43b700 1 --2- 172.21.15.78:0/2560276372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26780c6cd0 0x7f26780c70b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:36.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.874+0000 7f267d43b700 1 -- 172.21.15.78:0/2560276372 >> 172.21.15.78:0/2560276372 conn(0x7f26780abdf0 msgr2=0x7f26780ac1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:36.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.874+0000 7f267d43b700 1 -- 172.21.15.78:0/2560276372 shutdown_connections 2023-10-14T11:58:36.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.874+0000 7f267d43b700 1 -- 172.21.15.78:0/2560276372 wait complete. 2023-10-14T11:58:36.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.875+0000 7f267d43b700 1 Processor -- start 2023-10-14T11:58:36.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.875+0000 7f267d43b700 1 -- start start 2023-10-14T11:58:36.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.876+0000 7f267d43b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26780bd950 0x7f26780c03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:36.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.876+0000 7f267d43b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f26780c6cd0 0x7f26780c08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:36.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.876+0000 7f267d43b700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f26780c3650 0x7f26780c3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:36.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.876+0000 7f267d43b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26780be4b0 con 0x7f26780bd950 2023-10-14T11:58:36.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.876+0000 7f267d43b700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26781520b0 con 0x7f26780c6cd0 2023-10-14T11:58:36.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.876+0000 7f267d43b700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26781ea640 con 0x7f26780c3650 2023-10-14T11:58:36.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.876+0000 7f2676ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26780bd950 0x7f26780c03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:36.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.876+0000 7f2676ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26780bd950 0x7f26780c03b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45362/0 (socket says 172.21.15.78:45362) 2023-10-14T11:58:36.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.876+0000 7f2676ffd700 1 -- 172.21.15.78:0/706273948 learned_addr learned my addr 172.21.15.78:0/706273948 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:36.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.876+0000 7f26767fc700 1 --2- 172.21.15.78:0/706273948 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f26780c6cd0 0x7f26780c08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:36.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.876+0000 7f26777fe700 1 --2- 172.21.15.78:0/706273948 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f26780c3650 0x7f26780c3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:36.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.876+0000 7f2676ffd700 1 -- 172.21.15.78:0/706273948 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f26780c3650 msgr2=0x7f26780c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:36.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.876+0000 7f2676ffd700 1 --2- 172.21.15.78:0/706273948 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f26780c3650 0x7f26780c3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:36.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.876+0000 7f2676ffd700 1 -- 172.21.15.78:0/706273948 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f26780c6cd0 msgr2=0x7f26780c08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:36.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.876+0000 7f2676ffd700 1 --2- 172.21.15.78:0/706273948 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f26780c6cd0 0x7f26780c08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:36.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.876+0000 7f2676ffd700 1 -- 172.21.15.78:0/706273948 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2668009550 con 0x7f26780bd950 2023-10-14T11:58:36.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.876+0000 7f26767fc700 1 --2- 172.21.15.78:0/706273948 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f26780c6cd0 0x7f26780c08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:36.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.876+0000 7f2676ffd700 1 --2- 172.21.15.78:0/706273948 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26780bd950 0x7f26780c03b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2668009c10 tx=0x7f2668004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:36.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.876+0000 7f265f7fe700 1 -- 172.21.15.78:0/706273948 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2668003d80 con 0x7f26780bd950 2023-10-14T11:58:36.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.876+0000 7f265f7fe700 1 -- 172.21.15.78:0/706273948 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f266801f070 con 0x7f26780bd950 2023-10-14T11:58:36.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.876+0000 7f265f7fe700 1 -- 172.21.15.78:0/706273948 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2668011850 con 0x7f26780bd950 2023-10-14T11:58:36.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.876+0000 7f267d43b700 1 -- 172.21.15.78:0/706273948 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26781eab70 con 0x7f26780bd950 2023-10-14T11:58:36.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.876+0000 7f267d43b700 1 -- 172.21.15.78:0/706273948 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26780c0e30 con 0x7f26780bd950 2023-10-14T11:58:36.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.877+0000 7f267d43b700 1 -- 172.21.15.78:0/706273948 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26780683e0 con 0x7f26780bd950 2023-10-14T11:58:36.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.878+0000 7f265f7fe700 1 -- 172.21.15.78:0/706273948 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2668021030 con 0x7f26780bd950 2023-10-14T11:58:36.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.878+0000 7f265f7fe700 1 --2- 172.21.15.78:0/706273948 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f266406af00 0x7f266406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:36.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.878+0000 7f26767fc700 1 --2- 172.21.15.78:0/706273948 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f266406af00 0x7f266406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:36.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.878+0000 7f265f7fe700 1 -- 172.21.15.78:0/706273948 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f266808ab10 con 0x7f26780bd950 2023-10-14T11:58:36.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.879+0000 7f26767fc700 1 --2- 172.21.15.78:0/706273948 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f266406af00 0x7f266406d3c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f267814c150 tx=0x7f266c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:36.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:36.883+0000 7f265f7fe700 1 -- 172.21.15.78:0/706273948 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2668055c50 con 0x7f26780bd950 2023-10-14T11:58:37.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.075+0000 7f267d43b700 1 -- 172.21.15.78:0/706273948 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2678074b80 con 0x7f26780bd950 2023-10-14T11:58:37.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.076+0000 7f265f7fe700 1 -- 172.21.15.78:0/706273948 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2668029020 con 0x7f26780bd950 2023-10-14T11:58:37.308 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-10-14T11:58:37.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.078+0000 7f267d43b700 1 -- 172.21.15.78:0/706273948 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f266406af00 msgr2=0x7f266406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:37.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.078+0000 7f267d43b700 1 --2- 172.21.15.78:0/706273948 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f266406af00 0x7f266406d3c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f267814c150 tx=0x7f266c008040 comp rx=0 tx=0).stop 2023-10-14T11:58:37.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.078+0000 7f267d43b700 1 -- 172.21.15.78:0/706273948 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26780bd950 msgr2=0x7f26780c03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:37.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.078+0000 7f267d43b700 1 --2- 172.21.15.78:0/706273948 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26780bd950 0x7f26780c03b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2668009c10 tx=0x7f2668004750 comp rx=0 tx=0).stop 2023-10-14T11:58:37.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.078+0000 7f267d43b700 1 -- 172.21.15.78:0/706273948 shutdown_connections 2023-10-14T11:58:37.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.078+0000 7f267d43b700 1 --2- 172.21.15.78:0/706273948 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f26780c3650 0x7f26780c3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:37.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.078+0000 7f267d43b700 1 --2- 172.21.15.78:0/706273948 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26780bd950 0x7f26780c03b0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:37.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.078+0000 7f267d43b700 1 --2- 172.21.15.78:0/706273948 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f266406af00 0x7f266406d3c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:37.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.078+0000 7f267d43b700 1 --2- 172.21.15.78:0/706273948 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f26780c6cd0 0x7f26780c08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:37.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.078+0000 7f267d43b700 1 -- 172.21.15.78:0/706273948 >> 172.21.15.78:0/706273948 conn(0x7f26780abdf0 msgr2=0x7f267806b610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:37.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.078+0000 7f267d43b700 1 -- 172.21.15.78:0/706273948 shutdown_connections 2023-10-14T11:58:37.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.078+0000 7f267d43b700 1 -- 172.21.15.78:0/706273948 wait complete. 2023-10-14T11:58:37.313 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T11:58:37.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.448+0000 7fa2d5697700 1 Processor -- start 2023-10-14T11:58:37.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.448+0000 7fa2d5697700 1 -- start start 2023-10-14T11:58:37.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.448+0000 7fa2d5697700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa2d00ad550 0x7fa2d00c3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:37.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.448+0000 7fa2d5697700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2d00c4c10 con 0x7fa2d01500a0 2023-10-14T11:58:37.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.448+0000 7fa2d5697700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2d0070260 con 0x7fa2d00adaa0 2023-10-14T11:58:37.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.448+0000 7fa2d5697700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d0153420 con 0x7fa2d00ad550 2023-10-14T11:58:37.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.448+0000 7fa2ceffd700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fa2d01500a0 0x7fa2d00a9c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50246/0 (socket says 172.21.15.78:50246) 2023-10-14T11:58:37.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.448+0000 7fa2ceffd700 1 -- 172.21.15.78:0/322586346 learned_addr learned my addr 172.21.15.78:0/322586346 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:37.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.449+0000 7fa2ce7fc700 1 --2- 172.21.15.78:0/322586346 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa2d00ad550 0x7fa2d00c3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:37.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.449+0000 7fa2cdffb700 1 -- 172.21.15.78:0/322586346 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3754655626 0 0) 0x7fa2d00c4c10 con 0x7fa2d01500a0 2023-10-14T11:58:37.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.449+0000 7fa2cdffb700 1 -- 172.21.15.78:0/322586346 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2b80035b0 con 0x7fa2d01500a0 2023-10-14T11:58:37.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.449+0000 7fa2cdffb700 1 -- 172.21.15.78:0/322586346 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3815733062 0 0) 0x7fa2b80035b0 con 0x7fa2d01500a0 2023-10-14T11:58:37.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.449+0000 7fa2cdffb700 1 -- 172.21.15.78:0/322586346 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa2d00ad550 msgr2=0x7fa2d00c3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:37.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.449+0000 7fa2cdffb700 1 --2- 172.21.15.78:0/322586346 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa2d00ad550 0x7fa2d00c3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:37.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.449+0000 7fa2cdffb700 1 -- 172.21.15.78:0/322586346 >> v1:172.21.15.195:6789/0 conn(0x7fa2d00adaa0 legacy=0x7fa2d00c4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:37.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.449+0000 7fa2cdffb700 1 -- 172.21.15.78:0/322586346 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d0070500 con 0x7fa2d01500a0 2023-10-14T11:58:37.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.449+0000 7fa2cdffb700 1 -- 172.21.15.78:0/322586346 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441799917 0 0) 0x7fa2c0001670 con 0x7fa2d01500a0 2023-10-14T11:58:37.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.449+0000 7fa2cdffb700 1 -- 172.21.15.78:0/322586346 >> v1:172.21.15.78:6789/0 conn(0x7fa2d01500a0 legacy=0x7fa2d00a9c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:37.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.449+0000 7fa2cdffb700 1 --2- 172.21.15.78:0/322586346 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa2b8003a30 0x7fa2b8003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:37.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.449+0000 7fa2cdffb700 1 --2- 172.21.15.78:0/322586346 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa2d00ad550 0x7fa2b8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:37.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.449+0000 7fa2cdffb700 1 --2- 172.21.15.78:0/322586346 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2b8004d90 0x7fa2b8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:37.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.449+0000 7fa2cdffb700 1 -- 172.21.15.78:0/322586346 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d0153420 con 0x7fa2b8004d90 2023-10-14T11:58:37.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.449+0000 7fa2cdffb700 1 -- 172.21.15.78:0/322586346 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b800bac0 con 0x7fa2b8003a30 2023-10-14T11:58:37.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.449+0000 7fa2cdffb700 1 -- 172.21.15.78:0/322586346 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b800bc80 con 0x7fa2d00ad550 2023-10-14T11:58:37.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.449+0000 7fa2cf7fe700 1 --2- 172.21.15.78:0/322586346 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2b8004d90 0x7fa2b8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:37.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.449+0000 7fa2ceffd700 1 --2- 172.21.15.78:0/322586346 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa2b8003a30 0x7fa2b8003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:37.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.449+0000 7fa2ce7fc700 1 --2- 172.21.15.78:0/322586346 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa2d00ad550 0x7fa2b8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:37.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.449+0000 7fa2cf7fe700 1 -- 172.21.15.78:0/322586346 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa2d00ad550 msgr2=0x7fa2b8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:37.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.449+0000 7fa2cf7fe700 1 --2- 172.21.15.78:0/322586346 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa2d00ad550 0x7fa2b8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:37.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.449+0000 7fa2cf7fe700 1 -- 172.21.15.78:0/322586346 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa2b8003a30 msgr2=0x7fa2b8003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:37.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.449+0000 7fa2cf7fe700 1 --2- 172.21.15.78:0/322586346 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa2b8003a30 0x7fa2b8003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:37.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.449+0000 7fa2cf7fe700 1 -- 172.21.15.78:0/322586346 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b80035b0 con 0x7fa2b8004d90 2023-10-14T11:58:37.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.449+0000 7fa2ce7fc700 1 --2- 172.21.15.78:0/322586346 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa2d00ad550 0x7fa2b8004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:37.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.450+0000 7fa2cf7fe700 1 --2- 172.21.15.78:0/322586346 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2b8004d90 0x7fa2b8005210 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa2c800c210 tx=0x7fa2b8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62c1345469a1c9d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:37.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.450+0000 7fa2cdffb700 1 -- 172.21.15.78:0/322586346 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c8001df0 con 0x7fa2b8004d90 2023-10-14T11:58:37.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.450+0000 7fa2cdffb700 1 -- 172.21.15.78:0/322586346 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2c800cd70 con 0x7fa2b8004d90 2023-10-14T11:58:37.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.450+0000 7fa2cdffb700 1 -- 172.21.15.78:0/322586346 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c8003c40 con 0x7fa2b8004d90 2023-10-14T11:58:37.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.450+0000 7fa2d5697700 1 -- 172.21.15.78:0/322586346 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2b8004d90 msgr2=0x7fa2b8005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:37.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.450+0000 7fa2d5697700 1 --2- 172.21.15.78:0/322586346 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2b8004d90 0x7fa2b8005210 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa2c800c210 tx=0x7fa2b8005880 comp rx=0 tx=0).stop 2023-10-14T11:58:37.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.450+0000 7fa2cf7fe700 1 -- 172.21.15.78:0/322586346 reap_dead start 2023-10-14T11:58:37.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.450+0000 7fa2d5697700 1 -- 172.21.15.78:0/322586346 shutdown_connections 2023-10-14T11:58:37.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.450+0000 7fa2d5697700 1 -- 172.21.15.78:0/322586346 >> 172.21.15.78:0/322586346 conn(0x7fa2d0072a00 msgr2=0x7fa2d00733e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:37.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.450+0000 7fa2d5697700 1 -- 172.21.15.78:0/322586346 shutdown_connections 2023-10-14T11:58:37.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.450+0000 7fa2d5697700 1 -- 172.21.15.78:0/322586346 wait complete. 2023-10-14T11:58:37.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.452+0000 7fa2d5697700 1 Processor -- start 2023-10-14T11:58:37.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.452+0000 7fa2d5697700 1 -- start start 2023-10-14T11:58:37.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.452+0000 7fa2d5697700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa2b8004d90 0x7fa2d01ed1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:37.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.452+0000 7fa2d5697700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa2d00c8d80 0x7fa2d01ed710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:37.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.452+0000 7fa2d5697700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2d00bdf80 0x7fa2d00be410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:37.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.452+0000 7fa2d5697700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d006da50 con 0x7fa2d00bdf80 2023-10-14T11:58:37.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.452+0000 7fa2d5697700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d006ce60 con 0x7fa2b8004d90 2023-10-14T11:58:37.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.452+0000 7fa2d5697700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d006bac0 con 0x7fa2d00c8d80 2023-10-14T11:58:37.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.452+0000 7fa2cf7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2d00bdf80 0x7fa2d00be410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:37.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.452+0000 7fa2cf7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2d00bdf80 0x7fa2d00be410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45386/0 (socket says 172.21.15.78:45386) 2023-10-14T11:58:37.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.452+0000 7fa2cf7fe700 1 -- 172.21.15.78:0/701312235 learned_addr learned my addr 172.21.15.78:0/701312235 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:37.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.452+0000 7fa2ce7fc700 1 --2- 172.21.15.78:0/701312235 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa2d00c8d80 0x7fa2d01ed710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:37.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.452+0000 7fa2ceffd700 1 --2- 172.21.15.78:0/701312235 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa2b8004d90 0x7fa2d01ed1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:37.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.453+0000 7fa2cf7fe700 1 -- 172.21.15.78:0/701312235 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa2d00c8d80 msgr2=0x7fa2d01ed710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:37.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.453+0000 7fa2cf7fe700 1 --2- 172.21.15.78:0/701312235 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa2d00c8d80 0x7fa2d01ed710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:37.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.453+0000 7fa2cf7fe700 1 -- 172.21.15.78:0/701312235 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa2b8004d90 msgr2=0x7fa2d01ed1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:37.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.453+0000 7fa2cf7fe700 1 --2- 172.21.15.78:0/701312235 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa2b8004d90 0x7fa2d01ed1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:37.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.453+0000 7fa2cf7fe700 1 -- 172.21.15.78:0/701312235 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d01edc50 con 0x7fa2d00bdf80 2023-10-14T11:58:37.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.453+0000 7fa2cf7fe700 1 --2- 172.21.15.78:0/701312235 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2d00bdf80 0x7fa2d00be410 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8005040 tx=0x7fa2c80014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:37.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.453+0000 7fa2b77fe700 1 -- 172.21.15.78:0/701312235 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c800cee0 con 0x7fa2d00bdf80 2023-10-14T11:58:37.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.453+0000 7fa2b77fe700 1 -- 172.21.15.78:0/701312235 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2c8003e30 con 0x7fa2d00bdf80 2023-10-14T11:58:37.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.453+0000 7fa2d5697700 1 -- 172.21.15.78:0/701312235 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2d006d490 con 0x7fa2d00bdf80 2023-10-14T11:58:37.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.453+0000 7fa2b77fe700 1 -- 172.21.15.78:0/701312235 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c8005310 con 0x7fa2d00bdf80 2023-10-14T11:58:37.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.453+0000 7fa2d5697700 1 -- 172.21.15.78:0/701312235 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2d006c810 con 0x7fa2d00bdf80 2023-10-14T11:58:37.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.454+0000 7fa2d5697700 1 -- 172.21.15.78:0/701312235 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2d006c670 con 0x7fa2d00bdf80 2023-10-14T11:58:37.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.455+0000 7fa2b77fe700 1 -- 172.21.15.78:0/701312235 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa2c8019070 con 0x7fa2d00bdf80 2023-10-14T11:58:37.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.455+0000 7fa2b77fe700 1 --2- 172.21.15.78:0/701312235 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa2bc06af00 0x7fa2bc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:37.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.455+0000 7fa2b77fe700 1 -- 172.21.15.78:0/701312235 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa2c808a760 con 0x7fa2d00bdf80 2023-10-14T11:58:37.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.455+0000 7fa2ceffd700 1 --2- 172.21.15.78:0/701312235 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa2bc06af00 0x7fa2bc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:37.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.456+0000 7fa2ceffd700 1 --2- 172.21.15.78:0/701312235 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa2bc06af00 0x7fa2bc06d3c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa2c00011e0 tx=0x7fa2c0001150 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:37.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.459+0000 7fa2b77fe700 1 -- 172.21.15.78:0/701312235 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa2c80559a0 con 0x7fa2d00bdf80 2023-10-14T11:58:37.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.647+0000 7fa2d5697700 1 -- 172.21.15.78:0/701312235 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa2d006c670 con 0x7fa2d00bdf80 2023-10-14T11:58:37.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.648+0000 7fa2b77fe700 1 -- 172.21.15.78:0/701312235 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa2d006c670 con 0x7fa2d00bdf80 2023-10-14T11:58:37.650 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2023-10-14T11:58:37.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.650+0000 7fa2d5697700 1 -- 172.21.15.78:0/701312235 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa2bc06af00 msgr2=0x7fa2bc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:37.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.650+0000 7fa2d5697700 1 --2- 172.21.15.78:0/701312235 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa2bc06af00 0x7fa2bc06d3c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa2c00011e0 tx=0x7fa2c0001150 comp rx=0 tx=0).stop 2023-10-14T11:58:37.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.650+0000 7fa2d5697700 1 -- 172.21.15.78:0/701312235 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2d00bdf80 msgr2=0x7fa2d00be410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:37.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.650+0000 7fa2d5697700 1 --2- 172.21.15.78:0/701312235 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2d00bdf80 0x7fa2d00be410 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8005040 tx=0x7fa2c80014a0 comp rx=0 tx=0).stop 2023-10-14T11:58:37.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.650+0000 7fa2d5697700 1 -- 172.21.15.78:0/701312235 shutdown_connections 2023-10-14T11:58:37.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.650+0000 7fa2d5697700 1 --2- 172.21.15.78:0/701312235 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2d00bdf80 0x7fa2d00be410 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:37.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.650+0000 7fa2d5697700 1 --2- 172.21.15.78:0/701312235 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa2bc06af00 0x7fa2bc06d3c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:37.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.650+0000 7fa2d5697700 1 --2- 172.21.15.78:0/701312235 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa2b8004d90 0x7fa2d01ed1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:37.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.650+0000 7fa2d5697700 1 --2- 172.21.15.78:0/701312235 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa2d00c8d80 0x7fa2d01ed710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:37.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.650+0000 7fa2d5697700 1 -- 172.21.15.78:0/701312235 >> 172.21.15.78:0/701312235 conn(0x7fa2d0072a00 msgr2=0x7fa2d01528e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:37.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.650+0000 7fa2d5697700 1 -- 172.21.15.78:0/701312235 shutdown_connections 2023-10-14T11:58:37.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.650+0000 7fa2d5697700 1 -- 172.21.15.78:0/701312235 wait complete. 2023-10-14T11:58:37.666 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T11:58:37.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.793+0000 7fb5e9121700 1 Processor -- start 2023-10-14T11:58:37.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.793+0000 7fb5e9121700 1 -- start start 2023-10-14T11:58:37.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.794+0000 7fb5e9121700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb5e40c44c0 0x7fb5e40c7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:37.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.794+0000 7fb5e9121700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb5e40c72a0 0x7fb5e40c86e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:37.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.794+0000 7fb5e9121700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e4156b20 con 0x7fb5e40c8c20 2023-10-14T11:58:37.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.794+0000 7fb5e9121700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5e4155f10 con 0x7fb5e40c44c0 2023-10-14T11:58:37.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.794+0000 7fb5e9121700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e41561f0 con 0x7fb5e40c72a0 2023-10-14T11:58:37.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.794+0000 7fb5e259c700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb5e40c72a0 0x7fb5e40c86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:37.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.794+0000 7fb5e259c700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb5e40c72a0 0x7fb5e40c86e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.78:45064/0 (socket says 172.21.15.78:45064) 2023-10-14T11:58:37.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.794+0000 7fb5e259c700 1 -- 172.21.15.78:0/3097912335 learned_addr learned my addr 172.21.15.78:0/3097912335 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:37.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.794+0000 7fb5e2d9d700 1 --2- 172.21.15.78:0/3097912335 >> v2:172.21.15.195:3300/0 conn(0x7fb5e40c44c0 0x7fb5e40c7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:37.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.794+0000 7fb5e2d9d700 1 -- 172.21.15.78:0/3097912335 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb5e40c72a0 msgr2=0x7fb5e40c86e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:37.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.794+0000 7fb5e2d9d700 1 --2- 172.21.15.78:0/3097912335 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb5e40c72a0 0x7fb5e40c86e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:37.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.794+0000 7fb5e2d9d700 1 -- 172.21.15.78:0/3097912335 >> v1:172.21.15.195:6789/0 conn(0x7fb5e40c8c20 legacy=0x7fb5e40c2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:37.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.794+0000 7fb5e2d9d700 1 -- 172.21.15.78:0/3097912335 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e41564d0 con 0x7fb5e40c44c0 2023-10-14T11:58:37.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.794+0000 7fb5e259c700 1 --2- 172.21.15.78:0/3097912335 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb5e40c72a0 0x7fb5e40c86e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T11:58:37.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.794+0000 7fb5e1d9b700 1 -- 172.21.15.78:0/3097912335 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2905425359 0 0) 0x7fb5e4156b20 con 0x7fb5e40c8c20 2023-10-14T11:58:37.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.795+0000 7fb5e2d9d700 1 --2- 172.21.15.78:0/3097912335 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb5e40c44c0 0x7fb5e40c7a00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8009840 tx=0x7fb5d8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1434b648d084627b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:37.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.795+0000 7fb5e1d9b700 1 -- 172.21.15.78:0/3097912335 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d8011830 con 0x7fb5e40c44c0 2023-10-14T11:58:37.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.795+0000 7fb5e1d9b700 1 -- 172.21.15.78:0/3097912335 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d8011990 con 0x7fb5e40c44c0 2023-10-14T11:58:37.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.795+0000 7fb5e1d9b700 1 -- 172.21.15.78:0/3097912335 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d8011d20 con 0x7fb5e40c44c0 2023-10-14T11:58:37.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.795+0000 7fb5e9121700 1 -- 172.21.15.78:0/3097912335 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb5e40c44c0 msgr2=0x7fb5e40c7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:37.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.795+0000 7fb5e9121700 1 --2- 172.21.15.78:0/3097912335 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb5e40c44c0 0x7fb5e40c7a00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8009840 tx=0x7fb5d8009b50 comp rx=0 tx=0).stop 2023-10-14T11:58:37.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.795+0000 7fb5e9121700 1 -- 172.21.15.78:0/3097912335 shutdown_connections 2023-10-14T11:58:37.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.795+0000 7fb5e9121700 1 --2- 172.21.15.78:0/3097912335 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb5e40c44c0 0x7fb5e40c7a00 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:37.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.795+0000 7fb5e9121700 1 --2- 172.21.15.78:0/3097912335 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb5e40c72a0 0x7fb5e40c86e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:37.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.795+0000 7fb5e9121700 1 -- 172.21.15.78:0/3097912335 >> 172.21.15.78:0/3097912335 conn(0x7fb5e40737d0 msgr2=0x7fb5e4073be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:37.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.796+0000 7fb5e9121700 1 -- 172.21.15.78:0/3097912335 shutdown_connections 2023-10-14T11:58:37.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.796+0000 7fb5e9121700 1 -- 172.21.15.78:0/3097912335 wait complete. 2023-10-14T11:58:37.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.797+0000 7fb5e9121700 1 Processor -- start 2023-10-14T11:58:37.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.797+0000 7fb5e9121700 1 -- start start 2023-10-14T11:58:37.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.798+0000 7fb5e9121700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb5e40c44c0 0x7fb5e40c1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:37.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.798+0000 7fb5e9121700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb5e40c72a0 0x7fb5e41e8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:37.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.798+0000 7fb5e9121700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5e40c8c20 0x7fb5e40bbe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:37.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.798+0000 7fb5e9121700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e40c03b0 con 0x7fb5e40c8c20 2023-10-14T11:58:37.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.798+0000 7fb5e9121700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e40c0690 con 0x7fb5e40c72a0 2023-10-14T11:58:37.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.798+0000 7fb5e9121700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e40c0970 con 0x7fb5e40c44c0 2023-10-14T11:58:37.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.798+0000 7fb5e359e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5e40c8c20 0x7fb5e40bbe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:37.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.798+0000 7fb5e2d9d700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb5e40c44c0 0x7fb5e40c1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:37.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.798+0000 7fb5e359e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5e40c8c20 0x7fb5e40bbe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45400/0 (socket says 172.21.15.78:45400) 2023-10-14T11:58:37.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.798+0000 7fb5e359e700 1 -- 172.21.15.78:0/1568533777 learned_addr learned my addr 172.21.15.78:0/1568533777 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:37.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.798+0000 7fb5e259c700 1 --2- 172.21.15.78:0/1568533777 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb5e40c72a0 0x7fb5e41e8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:37.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.798+0000 7fb5e359e700 1 -- 172.21.15.78:0/1568533777 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb5e40c44c0 msgr2=0x7fb5e40c1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:37.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.798+0000 7fb5e359e700 1 --2- 172.21.15.78:0/1568533777 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb5e40c44c0 0x7fb5e40c1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:37.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.798+0000 7fb5e359e700 1 -- 172.21.15.78:0/1568533777 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb5e40c72a0 msgr2=0x7fb5e41e8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:37.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.798+0000 7fb5e359e700 1 --2- 172.21.15.78:0/1568533777 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb5e40c72a0 0x7fb5e41e8d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:37.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.798+0000 7fb5e359e700 1 -- 172.21.15.78:0/1568533777 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d4009550 con 0x7fb5e40c8c20 2023-10-14T11:58:37.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.798+0000 7fb5e359e700 1 --2- 172.21.15.78:0/1568533777 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5e40c8c20 0x7fb5e40bbe20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc00c7a0 tx=0x7fb5dc00cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:37.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.799+0000 7fb5cb7fe700 1 -- 172.21.15.78:0/1568533777 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc003e30 con 0x7fb5e40c8c20 2023-10-14T11:58:37.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.799+0000 7fb5cb7fe700 1 -- 172.21.15.78:0/1568533777 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5dc005070 con 0x7fb5e40c8c20 2023-10-14T11:58:37.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.799+0000 7fb5e9121700 1 -- 172.21.15.78:0/1568533777 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5d80094a0 con 0x7fb5e40c8c20 2023-10-14T11:58:37.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.799+0000 7fb5cb7fe700 1 -- 172.21.15.78:0/1568533777 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc005420 con 0x7fb5e40c8c20 2023-10-14T11:58:37.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.799+0000 7fb5e9121700 1 -- 172.21.15.78:0/1568533777 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5e40c0f30 con 0x7fb5e40c8c20 2023-10-14T11:58:37.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.800+0000 7fb5e9121700 1 -- 172.21.15.78:0/1568533777 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5e4074b80 con 0x7fb5e40c8c20 2023-10-14T11:58:37.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.800+0000 7fb5cb7fe700 1 -- 172.21.15.78:0/1568533777 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb5dc005580 con 0x7fb5e40c8c20 2023-10-14T11:58:37.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.800+0000 7fb5cb7fe700 1 --2- 172.21.15.78:0/1568533777 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb5d006af00 0x7fb5d006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:37.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.800+0000 7fb5cb7fe700 1 -- 172.21.15.78:0/1568533777 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb5dc088ae0 con 0x7fb5e40c8c20 2023-10-14T11:58:37.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.800+0000 7fb5e2d9d700 1 --2- 172.21.15.78:0/1568533777 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb5d006af00 0x7fb5d006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:37.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.801+0000 7fb5e2d9d700 1 --2- 172.21.15.78:0/1568533777 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb5d006af00 0x7fb5d006d3c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb5d80090b0 tx=0x7fb5d8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:37.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.805+0000 7fb5cb7fe700 1 -- 172.21.15.78:0/1568533777 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb5dc0576b0 con 0x7fb5e40c8c20 2023-10-14T11:58:37.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.993+0000 7fb5e9121700 1 -- 172.21.15.78:0/1568533777 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5e40c7080 con 0x7fb5e40c8c20 2023-10-14T11:58:37.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.995+0000 7fb5cb7fe700 1 -- 172.21.15.78:0/1568533777 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16481 (secure 0 0 0) 0x7fb5dc017070 con 0x7fb5e40c8c20 2023-10-14T11:58:37.997 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:37.997 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":21,"fsid":"4c5dc134-3092-406a-a1e1-13e6d9e926e8","created":"2023-10-14T11:57:57.363750+0000","modified":"2023-10-14T11:58:16.139413+0000","last_up_change":"2023-10-14T11:58:06.982472+0000","last_in_change":"2023-10-14T11:57:59.821373+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T11:58:09.001200+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T11:58:12.778573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T11:58:13.490286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"625a5fc0-9474-4eae-b80e-14c27a74ad19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6801","nonce":3884969780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6803","nonce":3884969780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6807","nonce":3884969780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6805","nonce":3884969780}]},"public_addr":"172.21.15.78:6801/3884969780","cluster_addr":"172.21.15.78:6803/3884969780","heartbeat_back_addr":"172.21.15.78:6807/3884969780","heartbeat_front_addr":"172.21.15.78:6805/3884969780","state":["exists","up"]},{"osd":1,"uuid":"49d63db1-2981-49ab-9c40-c50fc744b982","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6809","nonce":2176928406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6811","nonce":2176928406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6815","nonce":2176928406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6813","nonce":2176928406}]},"public_addr":"172.21.15.78:6809/2176928406","cluster_addr":"172.21.15.78:6811/2176928406","heartbeat_back_addr":"172.21.15.78:6815/2176928406","heartbeat_front_addr":"172.21.15.78:6813/2176928406","state":["exists","up"]},{"osd":2,"uuid":"1d7d0b29-cbd1-4717-9fe6-765a8d4998b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6825","nonce":3647044577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6827","nonce":3647044577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6831","nonce":3647044577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6829","nonce":3647044577}]},"public_addr":"172.21.15.78:6825/3647044577","cluster_addr":"172.21.15.78:6827/3647044577","heartbeat_back_addr":"172.21.15.78:6831/3647044577","heartbeat_front_addr":"172.21.15.78:6829/3647044577","state":["exists","up"]},{"osd":3,"uuid":"2f75ded1-e9e2-40e7-99ae-45e1e425e1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6817","nonce":3584112288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6819","nonce":3584112288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6823","nonce":3584112288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6821","nonce":3584112288}]},"public_addr":"172.21.15.78:6817/3584112288","cluster_addr":"172.21.15.78:6819/3584112288","heartbeat_back_addr":"172.21.15.78:6823/3584112288","heartbeat_front_addr":"172.21.15.78:6821/3584112288","state":["exists","up"]},{"osd":4,"uuid":"5130b432-8358-46cb-8a31-3064c976f6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6801","nonce":3732756301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6803","nonce":3732756301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6807","nonce":3732756301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6805","nonce":3732756301}]},"public_addr":"172.21.15.195:6801/3732756301","cluster_addr":"172.21.15.195:6803/3732756301","heartbeat_back_addr":"172.21.15.195:6807/3732756301","heartbeat_front_addr":"172.21.15.195:6805/3732756301","state":["exists","up"]},{"osd":5,"uuid":"60e234a9-a506-4e40-9c19-49a58f3cf28c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6825","nonce":2857706411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6827","nonce":2857706411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6831","nonce":2857706411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6829","nonce":2857706411}]},"public_addr":"172.21.15.195:6825/2857706411","cluster_addr":"172.21.15.195:6827/2857706411","heartbeat_back_addr":"172.21.15.195:6831/2857706411","heartbeat_front_addr":"172.21.15.195:6829/2857706411","state":["exists","up"]},{"osd":6,"uuid":"df7d2dd6-26b6-4eb1-96e5-94d1f3955b33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6810","nonce":72554374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6813","nonce":72554374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6820","nonce":72554374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6817","nonce":72554374}]},"public_addr":"172.21.15.195:6810/72554374","cluster_addr":"172.21.15.195:6813/72554374","heartbeat_back_addr":"172.21.15.195:6820/72554374","heartbeat_front_addr":"172.21.15.195:6817/72554374","state":["exists","up"]},{"osd":7,"uuid":"47a9d4de-f56d-4aa4-9900-c4a9cf9b2f97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6809","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6811","nonce":482472979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6816","nonce":482472979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6823","nonce":482472979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6821","nonce":482472979}]},"public_addr":"172.21.15.195:6811/482472979","cluster_addr":"172.21.15.195:6816/482472979","heartbeat_back_addr":"172.21.15.195:6823/482472979","heartbeat_front_addr":"172.21.15.195:6821/482472979","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.754003+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.005994+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.043603+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.024506+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.062585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.033767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.994825+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.038300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T11:58:37.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.997+0000 7fb5e9121700 1 -- 172.21.15.78:0/1568533777 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb5d006af00 msgr2=0x7fb5d006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:37.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.997+0000 7fb5e9121700 1 --2- 172.21.15.78:0/1568533777 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb5d006af00 0x7fb5d006d3c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb5d80090b0 tx=0x7fb5d8008030 comp rx=0 tx=0).stop 2023-10-14T11:58:38.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.997+0000 7fb5e9121700 1 -- 172.21.15.78:0/1568533777 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5e40c8c20 msgr2=0x7fb5e40bbe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:38.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.997+0000 7fb5e9121700 1 --2- 172.21.15.78:0/1568533777 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5e40c8c20 0x7fb5e40bbe20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc00c7a0 tx=0x7fb5dc00cab0 comp rx=0 tx=0).stop 2023-10-14T11:58:38.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.997+0000 7fb5e9121700 1 -- 172.21.15.78:0/1568533777 shutdown_connections 2023-10-14T11:58:38.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.997+0000 7fb5e9121700 1 --2- 172.21.15.78:0/1568533777 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5e40c8c20 0x7fb5e40bbe20 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:38.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.997+0000 7fb5e9121700 1 --2- 172.21.15.78:0/1568533777 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb5e40c44c0 0x7fb5e40c1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:38.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.997+0000 7fb5e9121700 1 --2- 172.21.15.78:0/1568533777 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb5d006af00 0x7fb5d006d3c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:38.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.997+0000 7fb5e9121700 1 --2- 172.21.15.78:0/1568533777 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb5e40c72a0 0x7fb5e41e8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:38.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.997+0000 7fb5e9121700 1 -- 172.21.15.78:0/1568533777 >> 172.21.15.78:0/1568533777 conn(0x7fb5e40737d0 msgr2=0x7fb5e40c7780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:38.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.997+0000 7fb5e9121700 1 -- 172.21.15.78:0/1568533777 shutdown_connections 2023-10-14T11:58:38.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:37.997+0000 7fb5e9121700 1 -- 172.21.15.78:0/1568533777 wait complete. 2023-10-14T11:58:38.011 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T11:58:09.001200+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T11:58:12.778573+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T11:58:13.490286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T11:58:38.012 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T11:58:38.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.137+0000 7f277f2c7700 1 Processor -- start 2023-10-14T11:58:38.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.137+0000 7f277f2c7700 1 -- start start 2023-10-14T11:58:38.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.138+0000 7f277f2c7700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f277813dbe0 0x7f277806a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:38.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.138+0000 7f277f2c7700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f27780c4540 0x7f27780c02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:38.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.138+0000 7f277f2c7700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f277806ebe0 con 0x7f277806ae10 2023-10-14T11:58:38.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.138+0000 7f277f2c7700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f277806d330 con 0x7f27780c4540 2023-10-14T11:58:38.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.138+0000 7f277f2c7700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27781558c0 con 0x7f277813dbe0 2023-10-14T11:58:38.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.138+0000 7f277c862700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f27780c4540 0x7f27780c02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:38.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.138+0000 7f277c862700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f27780c4540 0x7f27780c02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45404/0 (socket says 172.21.15.78:45404) 2023-10-14T11:58:38.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.138+0000 7f277c862700 1 -- 172.21.15.78:0/24159255 learned_addr learned my addr 172.21.15.78:0/24159255 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:38.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.138+0000 7f277d063700 1 --2- 172.21.15.78:0/24159255 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f277813dbe0 0x7f277806a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:38.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.138+0000 7f277c862700 1 -- 172.21.15.78:0/24159255 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f277813dbe0 msgr2=0x7f277806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:38.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.138+0000 7f277c862700 1 --2- 172.21.15.78:0/24159255 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f277813dbe0 0x7f277806a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:38.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.138+0000 7f277c862700 1 -- 172.21.15.78:0/24159255 >> v1:172.21.15.195:6789/0 conn(0x7f277806ae10 legacy=0x7f27780c0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:38.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.138+0000 7f277c862700 1 -- 172.21.15.78:0/24159255 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2778154b00 con 0x7f27780c4540 2023-10-14T11:58:38.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.139+0000 7f277c862700 1 --2- 172.21.15.78:0/24159255 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27780c4540 0x7f27780c02e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f27740098f0 tx=0x7f2774009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fd4e79bc6107b75 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:38.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.139+0000 7f276ffff700 1 -- 172.21.15.78:0/24159255 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277400b8a0 con 0x7f27780c4540 2023-10-14T11:58:38.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.139+0000 7f276ffff700 1 -- 172.21.15.78:0/24159255 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f277400ba00 con 0x7f27780c4540 2023-10-14T11:58:38.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.139+0000 7f276ffff700 1 -- 172.21.15.78:0/24159255 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277400bd90 con 0x7f27780c4540 2023-10-14T11:58:38.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.139+0000 7f277f2c7700 1 -- 172.21.15.78:0/24159255 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27780c4540 msgr2=0x7f27780c02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:38.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.139+0000 7f277f2c7700 1 --2- 172.21.15.78:0/24159255 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27780c4540 0x7f27780c02e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f27740098f0 tx=0x7f2774009c00 comp rx=0 tx=0).stop 2023-10-14T11:58:38.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.139+0000 7f277f2c7700 1 -- 172.21.15.78:0/24159255 shutdown_connections 2023-10-14T11:58:38.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.139+0000 7f277f2c7700 1 --2- 172.21.15.78:0/24159255 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f277813dbe0 0x7f277806a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:38.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.139+0000 7f277f2c7700 1 --2- 172.21.15.78:0/24159255 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27780c4540 0x7f27780c02e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:38.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.139+0000 7f277f2c7700 1 -- 172.21.15.78:0/24159255 >> 172.21.15.78:0/24159255 conn(0x7f27780c3a50 msgr2=0x7f27780c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:38.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.139+0000 7f277f2c7700 1 -- 172.21.15.78:0/24159255 shutdown_connections 2023-10-14T11:58:38.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.139+0000 7f277f2c7700 1 -- 172.21.15.78:0/24159255 wait complete. 2023-10-14T11:58:38.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.140+0000 7f277f2c7700 1 Processor -- start 2023-10-14T11:58:38.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.140+0000 7f277f2c7700 1 -- start start 2023-10-14T11:58:38.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.141+0000 7f277f2c7700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f277806ae10 0x7f2778137d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:38.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.141+0000 7f277f2c7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f27780c4540 0x7f2778139740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:38.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.141+0000 7f277f2c7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f277813dbe0 0x7f2778156140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:38.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.141+0000 7f277f2c7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27781ea940 con 0x7f277813dbe0 2023-10-14T11:58:38.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.141+0000 7f277f2c7700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2778138250 con 0x7f27780c4540 2023-10-14T11:58:38.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.141+0000 7f277f2c7700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2778155cf0 con 0x7f277806ae10 2023-10-14T11:58:38.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.141+0000 7f277d864700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f277813dbe0 0x7f2778156140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:38.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.141+0000 7f277d864700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f277813dbe0 0x7f2778156140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45412/0 (socket says 172.21.15.78:45412) 2023-10-14T11:58:38.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.141+0000 7f277d864700 1 -- 172.21.15.78:0/465174526 learned_addr learned my addr 172.21.15.78:0/465174526 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:38.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.141+0000 7f277d063700 1 --2- 172.21.15.78:0/465174526 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f277806ae10 0x7f2778137d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:38.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.141+0000 7f277c862700 1 --2- 172.21.15.78:0/465174526 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f27780c4540 0x7f2778139740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:38.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.141+0000 7f277d864700 1 -- 172.21.15.78:0/465174526 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f277806ae10 msgr2=0x7f2778137d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:38.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.141+0000 7f277d864700 1 --2- 172.21.15.78:0/465174526 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f277806ae10 0x7f2778137d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:38.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.141+0000 7f277d864700 1 -- 172.21.15.78:0/465174526 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f27780c4540 msgr2=0x7f2778139740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:38.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.141+0000 7f277d864700 1 --2- 172.21.15.78:0/465174526 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f27780c4540 0x7f2778139740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:38.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.141+0000 7f277d864700 1 -- 172.21.15.78:0/465174526 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2774009550 con 0x7f277813dbe0 2023-10-14T11:58:38.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.142+0000 7f277d864700 1 --2- 172.21.15.78:0/465174526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f277813dbe0 0x7f2778156140 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f277000cb40 tx=0x7f277000ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:38.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.142+0000 7f276dffb700 1 -- 172.21.15.78:0/465174526 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2770003c00 con 0x7f277813dbe0 2023-10-14T11:58:38.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.142+0000 7f276dffb700 1 -- 172.21.15.78:0/465174526 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2770003d60 con 0x7f277813dbe0 2023-10-14T11:58:38.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.142+0000 7f277f2c7700 1 -- 172.21.15.78:0/465174526 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27780a8630 con 0x7f277813dbe0 2023-10-14T11:58:38.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.142+0000 7f276dffb700 1 -- 172.21.15.78:0/465174526 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277000aa20 con 0x7f277813dbe0 2023-10-14T11:58:38.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.142+0000 7f277f2c7700 1 -- 172.21.15.78:0/465174526 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27780a8bf0 con 0x7f277813dbe0 2023-10-14T11:58:38.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.143+0000 7f277f2c7700 1 -- 172.21.15.78:0/465174526 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2778155e60 con 0x7f277813dbe0 2023-10-14T11:58:38.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.143+0000 7f276dffb700 1 -- 172.21.15.78:0/465174526 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f277000ab80 con 0x7f277813dbe0 2023-10-14T11:58:38.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.144+0000 7f276dffb700 1 --2- 172.21.15.78:0/465174526 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f275c06af00 0x7f275c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:38.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.144+0000 7f276dffb700 1 -- 172.21.15.78:0/465174526 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2770089400 con 0x7f277813dbe0 2023-10-14T11:58:38.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.144+0000 7f277d063700 1 --2- 172.21.15.78:0/465174526 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f275c06af00 0x7f275c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:38.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.144+0000 7f277d063700 1 --2- 172.21.15.78:0/465174526 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f275c06af00 0x7f275c06d3c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2768006b20 tx=0x7f2768008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:38.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.149+0000 7f276dffb700 1 -- 172.21.15.78:0/465174526 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2778155e60 con 0x7f277813dbe0 2023-10-14T11:58:38.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.336+0000 7f277f2c7700 1 -- 172.21.15.78:0/465174526 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2778155e60 con 0x7f277813dbe0 2023-10-14T11:58:38.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.337+0000 7f276dffb700 1 -- 172.21.15.78:0/465174526 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2778155e60 con 0x7f277813dbe0 2023-10-14T11:58:38.339 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-10-14T11:58:38.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.339+0000 7f277f2c7700 1 -- 172.21.15.78:0/465174526 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f275c06af00 msgr2=0x7f275c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:38.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.339+0000 7f277f2c7700 1 --2- 172.21.15.78:0/465174526 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f275c06af00 0x7f275c06d3c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2768006b20 tx=0x7f2768008040 comp rx=0 tx=0).stop 2023-10-14T11:58:38.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.339+0000 7f277f2c7700 1 -- 172.21.15.78:0/465174526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f277813dbe0 msgr2=0x7f2778156140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:38.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.339+0000 7f277f2c7700 1 --2- 172.21.15.78:0/465174526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f277813dbe0 0x7f2778156140 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f277000cb40 tx=0x7f277000ce50 comp rx=0 tx=0).stop 2023-10-14T11:58:38.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.339+0000 7f277f2c7700 1 -- 172.21.15.78:0/465174526 shutdown_connections 2023-10-14T11:58:38.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.339+0000 7f277f2c7700 1 --2- 172.21.15.78:0/465174526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f277813dbe0 0x7f2778156140 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:38.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.339+0000 7f277f2c7700 1 --2- 172.21.15.78:0/465174526 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f277806ae10 0x7f2778137d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:38.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.339+0000 7f277f2c7700 1 --2- 172.21.15.78:0/465174526 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f275c06af00 0x7f275c06d3c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:38.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.339+0000 7f277f2c7700 1 --2- 172.21.15.78:0/465174526 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f27780c4540 0x7f2778139740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:38.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.339+0000 7f277f2c7700 1 -- 172.21.15.78:0/465174526 >> 172.21.15.78:0/465174526 conn(0x7f27780c3a50 msgr2=0x7f27780c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:38.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.339+0000 7f277f2c7700 1 -- 172.21.15.78:0/465174526 shutdown_connections 2023-10-14T11:58:38.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.339+0000 7f277f2c7700 1 -- 172.21.15.78:0/465174526 wait complete. 2023-10-14T11:58:38.353 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T11:58:38.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.474+0000 7fb36f92d700 1 Processor -- start 2023-10-14T11:58:38.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.475+0000 7fb36f92d700 1 -- start start 2023-10-14T11:58:38.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.475+0000 7fb36f92d700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb3680c4540 0x7fb3680c02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:38.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.475+0000 7fb36f92d700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb36806ebe0 con 0x7fb36813dbe0 2023-10-14T11:58:38.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.475+0000 7fb36f92d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3681558c0 con 0x7fb36806b500 2023-10-14T11:58:38.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.475+0000 7fb36f92d700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb36806d330 con 0x7fb3680c4540 2023-10-14T11:58:38.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.476+0000 7fb36d6c9700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb36813dbe0 0x7fb36806a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50294/0 (socket says 172.21.15.78:50294) 2023-10-14T11:58:38.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.476+0000 7fb36d6c9700 1 -- 172.21.15.78:0/747238762 learned_addr learned my addr 172.21.15.78:0/747238762 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:38.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.476+0000 7fb36cec8700 1 --2- 172.21.15.78:0/747238762 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb3680c4540 0x7fb3680c02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:38.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.476+0000 7fb367fff700 1 -- 172.21.15.78:0/747238762 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 684380074 0 0) 0x7fb36806ebe0 con 0x7fb36813dbe0 2023-10-14T11:58:38.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.476+0000 7fb367fff700 1 -- 172.21.15.78:0/747238762 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3500035b0 con 0x7fb36813dbe0 2023-10-14T11:58:38.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.476+0000 7fb367fff700 1 -- 172.21.15.78:0/747238762 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3382621948 0 0) 0x7fb3500035b0 con 0x7fb36813dbe0 2023-10-14T11:58:38.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.476+0000 7fb367fff700 1 -- 172.21.15.78:0/747238762 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb3680c4540 msgr2=0x7fb3680c02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:38.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.476+0000 7fb367fff700 1 --2- 172.21.15.78:0/747238762 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb3680c4540 0x7fb3680c02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:38.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.476+0000 7fb367fff700 1 -- 172.21.15.78:0/747238762 >> v1:172.21.15.195:6789/0 conn(0x7fb36806b500 legacy=0x7fb3680c0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:38.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.476+0000 7fb367fff700 1 -- 172.21.15.78:0/747238762 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb368154b00 con 0x7fb36813dbe0 2023-10-14T11:58:38.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.476+0000 7fb367fff700 1 -- 172.21.15.78:0/747238762 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441799917 0 0) 0x7fb360001670 con 0x7fb36813dbe0 2023-10-14T11:58:38.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.476+0000 7fb367fff700 1 -- 172.21.15.78:0/747238762 >> v1:172.21.15.78:6789/0 conn(0x7fb36813dbe0 legacy=0x7fb36806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:38.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.476+0000 7fb367fff700 1 --2- 172.21.15.78:0/747238762 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb350003a30 0x7fb350003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:38.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.476+0000 7fb367fff700 1 --2- 172.21.15.78:0/747238762 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb350004850 0x7fb350004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:38.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.476+0000 7fb367fff700 1 --2- 172.21.15.78:0/747238762 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb3680c4540 0x7fb350005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:38.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.476+0000 7fb367fff700 1 -- 172.21.15.78:0/747238762 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36806d330 con 0x7fb350004850 2023-10-14T11:58:38.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.476+0000 7fb367fff700 1 -- 172.21.15.78:0/747238762 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35000bac0 con 0x7fb350003a30 2023-10-14T11:58:38.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.476+0000 7fb367fff700 1 -- 172.21.15.78:0/747238762 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb35000bc80 con 0x7fb3680c4540 2023-10-14T11:58:38.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.476+0000 7fb36cec8700 1 --2- 172.21.15.78:0/747238762 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb350004850 0x7fb350004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:38.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.476+0000 7fb36deca700 1 --2- 172.21.15.78:0/747238762 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb3680c4540 0x7fb350005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:38.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.476+0000 7fb36d6c9700 1 --2- 172.21.15.78:0/747238762 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb350003a30 0x7fb350003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:38.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.477+0000 7fb36cec8700 1 -- 172.21.15.78:0/747238762 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb3680c4540 msgr2=0x7fb350005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:38.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.477+0000 7fb36cec8700 1 --2- 172.21.15.78:0/747238762 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb3680c4540 0x7fb350005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:38.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.477+0000 7fb36cec8700 1 -- 172.21.15.78:0/747238762 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb350003a30 msgr2=0x7fb350003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:38.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.477+0000 7fb36cec8700 1 --2- 172.21.15.78:0/747238762 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb350003a30 0x7fb350003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:38.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.477+0000 7fb36cec8700 1 -- 172.21.15.78:0/747238762 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3500035b0 con 0x7fb350004850 2023-10-14T11:58:38.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.477+0000 7fb36deca700 1 --2- 172.21.15.78:0/747238762 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb3680c4540 0x7fb350005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:38.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.477+0000 7fb36d6c9700 1 --2- 172.21.15.78:0/747238762 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb350003a30 0x7fb350003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:38.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.477+0000 7fb36cec8700 1 --2- 172.21.15.78:0/747238762 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb350004850 0x7fb350004cd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb3680c4980 tx=0x7fb350005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36599f151250761e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:38.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.477+0000 7fb367fff700 1 -- 172.21.15.78:0/747238762 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35c00e6c0 con 0x7fb350004850 2023-10-14T11:58:38.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.477+0000 7fb367fff700 1 -- 172.21.15.78:0/747238762 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb35c00e820 con 0x7fb350004850 2023-10-14T11:58:38.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.477+0000 7fb367fff700 1 -- 172.21.15.78:0/747238762 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35c00ebb0 con 0x7fb350004850 2023-10-14T11:58:38.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.477+0000 7fb36f92d700 1 -- 172.21.15.78:0/747238762 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb350004850 msgr2=0x7fb350004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:38.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.477+0000 7fb36f92d700 1 --2- 172.21.15.78:0/747238762 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb350004850 0x7fb350004cd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb3680c4980 tx=0x7fb350005880 comp rx=0 tx=0).stop 2023-10-14T11:58:38.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.477+0000 7fb36deca700 1 -- 172.21.15.78:0/747238762 reap_dead start 2023-10-14T11:58:38.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.477+0000 7fb36f92d700 1 -- 172.21.15.78:0/747238762 shutdown_connections 2023-10-14T11:58:38.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.477+0000 7fb36f92d700 1 -- 172.21.15.78:0/747238762 >> 172.21.15.78:0/747238762 conn(0x7fb3680c3a50 msgr2=0x7fb3680c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:38.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.477+0000 7fb36f92d700 1 -- 172.21.15.78:0/747238762 shutdown_connections 2023-10-14T11:58:38.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.477+0000 7fb36f92d700 1 -- 172.21.15.78:0/747238762 wait complete. 2023-10-14T11:58:38.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.479+0000 7fb36f92d700 1 Processor -- start 2023-10-14T11:58:38.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.479+0000 7fb36f92d700 1 -- start start 2023-10-14T11:58:38.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.479+0000 7fb36f92d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb350004850 0x7fb3680abd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:38.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.479+0000 7fb36f92d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3680c0d20 0x7fb3681e9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:38.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.479+0000 7fb36f92d700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb3681e9f70 0x7fb368137550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:38.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.479+0000 7fb36f92d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36813d280 con 0x7fb3680c0d20 2023-10-14T11:58:38.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.479+0000 7fb36f92d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3681ecfa0 con 0x7fb350004850 2023-10-14T11:58:38.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.479+0000 7fb36f92d700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3681ed840 con 0x7fb3681e9f70 2023-10-14T11:58:38.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.479+0000 7fb36cec8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3680c0d20 0x7fb3681e9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:38.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.479+0000 7fb36cec8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3680c0d20 0x7fb3681e9a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45432/0 (socket says 172.21.15.78:45432) 2023-10-14T11:58:38.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.479+0000 7fb36cec8700 1 -- 172.21.15.78:0/3735765016 learned_addr learned my addr 172.21.15.78:0/3735765016 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:38.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.480+0000 7fb36d6c9700 1 --2- 172.21.15.78:0/3735765016 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb350004850 0x7fb3680abd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:38.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.480+0000 7fb36cec8700 1 -- 172.21.15.78:0/3735765016 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb3681e9f70 msgr2=0x7fb368137550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:38.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.480+0000 7fb36deca700 1 --2- 172.21.15.78:0/3735765016 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb3681e9f70 0x7fb368137550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:38.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.480+0000 7fb36cec8700 1 --2- 172.21.15.78:0/3735765016 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb3681e9f70 0x7fb368137550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:38.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.480+0000 7fb36cec8700 1 -- 172.21.15.78:0/3735765016 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb350004850 msgr2=0x7fb3680abd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:38.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.480+0000 7fb36cec8700 1 --2- 172.21.15.78:0/3735765016 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb350004850 0x7fb3680abd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:38.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.480+0000 7fb36cec8700 1 -- 172.21.15.78:0/3735765016 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3681ea3f0 con 0x7fb3680c0d20 2023-10-14T11:58:38.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.480+0000 7fb36d6c9700 1 --2- 172.21.15.78:0/3735765016 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb350004850 0x7fb3680abd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:38.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.480+0000 7fb36cec8700 1 --2- 172.21.15.78:0/3735765016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3680c0d20 0x7fb3681e9a30 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb3500047b0 tx=0x7fb35c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:38.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.480+0000 7fb365ffb700 1 -- 172.21.15.78:0/3735765016 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35c021070 con 0x7fb3680c0d20 2023-10-14T11:58:38.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.480+0000 7fb365ffb700 1 -- 172.21.15.78:0/3735765016 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb35c004510 con 0x7fb3680c0d20 2023-10-14T11:58:38.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.480+0000 7fb36f92d700 1 -- 172.21.15.78:0/3735765016 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3680ac310 con 0x7fb3680c0d20 2023-10-14T11:58:38.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.480+0000 7fb365ffb700 1 -- 172.21.15.78:0/3735765016 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35c00e6c0 con 0x7fb3680c0d20 2023-10-14T11:58:38.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.480+0000 7fb36f92d700 1 -- 172.21.15.78:0/3735765016 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3681ed4d0 con 0x7fb3680c0d20 2023-10-14T11:58:38.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.481+0000 7fb36f92d700 1 -- 172.21.15.78:0/3735765016 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb36813d280 con 0x7fb3680c0d20 2023-10-14T11:58:38.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.481+0000 7fb365ffb700 1 -- 172.21.15.78:0/3735765016 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb35c025070 con 0x7fb3680c0d20 2023-10-14T11:58:38.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.482+0000 7fb365ffb700 1 --2- 172.21.15.78:0/3735765016 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb35406af00 0x7fb35406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:38.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.482+0000 7fb365ffb700 1 -- 172.21.15.78:0/3735765016 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb35c08a910 con 0x7fb3680c0d20 2023-10-14T11:58:38.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.482+0000 7fb36d6c9700 1 --2- 172.21.15.78:0/3735765016 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb35406af00 0x7fb35406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:38.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.482+0000 7fb36d6c9700 1 --2- 172.21.15.78:0/3735765016 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb35406af00 0x7fb35406d3c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb360005940 tx=0x7fb360001080 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:38.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.486+0000 7fb365ffb700 1 -- 172.21.15.78:0/3735765016 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb36813d280 con 0x7fb3680c0d20 2023-10-14T11:58:38.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.675+0000 7fb36f92d700 1 -- 172.21.15.78:0/3735765016 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb36813d280 con 0x7fb3680c0d20 2023-10-14T11:58:38.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.676+0000 7fb365ffb700 1 -- 172.21.15.78:0/3735765016 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb35c02c020 con 0x7fb3680c0d20 2023-10-14T11:58:38.678 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-10-14T11:58:38.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.678+0000 7fb36f92d700 1 -- 172.21.15.78:0/3735765016 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb35406af00 msgr2=0x7fb35406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:38.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.678+0000 7fb36f92d700 1 --2- 172.21.15.78:0/3735765016 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb35406af00 0x7fb35406d3c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb360005940 tx=0x7fb360001080 comp rx=0 tx=0).stop 2023-10-14T11:58:38.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.678+0000 7fb36f92d700 1 -- 172.21.15.78:0/3735765016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3680c0d20 msgr2=0x7fb3681e9a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:38.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.678+0000 7fb36f92d700 1 --2- 172.21.15.78:0/3735765016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3680c0d20 0x7fb3681e9a30 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb3500047b0 tx=0x7fb35c0047c0 comp rx=0 tx=0).stop 2023-10-14T11:58:38.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.678+0000 7fb36f92d700 1 -- 172.21.15.78:0/3735765016 shutdown_connections 2023-10-14T11:58:38.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.678+0000 7fb36f92d700 1 --2- 172.21.15.78:0/3735765016 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb3681e9f70 0x7fb368137550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:38.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.678+0000 7fb36f92d700 1 --2- 172.21.15.78:0/3735765016 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb35406af00 0x7fb35406d3c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:38.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.678+0000 7fb36f92d700 1 --2- 172.21.15.78:0/3735765016 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb350004850 0x7fb3680abd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:38.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.678+0000 7fb36f92d700 1 --2- 172.21.15.78:0/3735765016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3680c0d20 0x7fb3681e9a30 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:38.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.678+0000 7fb36f92d700 1 -- 172.21.15.78:0/3735765016 >> 172.21.15.78:0/3735765016 conn(0x7fb3680c3a50 msgr2=0x7fb36806a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:38.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.678+0000 7fb36f92d700 1 -- 172.21.15.78:0/3735765016 shutdown_connections 2023-10-14T11:58:38.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.678+0000 7fb36f92d700 1 -- 172.21.15.78:0/3735765016 wait complete. 2023-10-14T11:58:38.692 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T11:58:38.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.813+0000 7f9def38c700 1 Processor -- start 2023-10-14T11:58:38.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.813+0000 7f9def38c700 1 -- start start 2023-10-14T11:58:38.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.814+0000 7f9def38c700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9de813dbe0 0x7f9de806a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:38.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.814+0000 7f9def38c700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9de80c4540 0x7f9de80c02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:38.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.814+0000 7f9def38c700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de806ebe0 con 0x7f9de806ae10 2023-10-14T11:58:38.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.814+0000 7f9def38c700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9de806d330 con 0x7f9de80c4540 2023-10-14T11:58:38.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.814+0000 7f9def38c700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9de81558c0 con 0x7f9de813dbe0 2023-10-14T11:58:38.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.814+0000 7f9dec927700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9de80c4540 0x7f9de80c02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:38.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.814+0000 7f9dec927700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9de80c4540 0x7f9de80c02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45440/0 (socket says 172.21.15.78:45440) 2023-10-14T11:58:38.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.814+0000 7f9dec927700 1 -- 172.21.15.78:0/3007831371 learned_addr learned my addr 172.21.15.78:0/3007831371 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:38.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.814+0000 7f9ded929700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f9de806ae10 0x7f9de80c0820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50316/0 (socket says 172.21.15.78:50316) 2023-10-14T11:58:38.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.814+0000 7f9ded128700 1 --2- 172.21.15.78:0/3007831371 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9de813dbe0 0x7f9de806a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:38.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.814+0000 7f9dec927700 1 -- 172.21.15.78:0/3007831371 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9de813dbe0 msgr2=0x7f9de806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:38.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.814+0000 7f9dec927700 1 --2- 172.21.15.78:0/3007831371 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9de813dbe0 0x7f9de806a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:38.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.814+0000 7f9ddffff700 1 -- 172.21.15.78:0/3007831371 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 338232914 0 0) 0x7f9de806ebe0 con 0x7f9de806ae10 2023-10-14T11:58:38.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.814+0000 7f9dec927700 1 -- 172.21.15.78:0/3007831371 >> v1:172.21.15.78:6789/0 conn(0x7f9de806ae10 legacy=0x7f9de80c0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:38.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.815+0000 7f9dec927700 1 -- 172.21.15.78:0/3007831371 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de8154b00 con 0x7f9de80c4540 2023-10-14T11:58:38.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.815+0000 7f9dec927700 1 --2- 172.21.15.78:0/3007831371 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9de80c4540 0x7f9de80c02e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9de40098f0 tx=0x7f9de4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26aa8a49f210825a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:38.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.815+0000 7f9ddffff700 1 -- 172.21.15.78:0/3007831371 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de400b8a0 con 0x7f9de80c4540 2023-10-14T11:58:38.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.815+0000 7f9ddffff700 1 -- 172.21.15.78:0/3007831371 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de400ba00 con 0x7f9de80c4540 2023-10-14T11:58:38.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.815+0000 7f9ddffff700 1 -- 172.21.15.78:0/3007831371 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de400bd90 con 0x7f9de80c4540 2023-10-14T11:58:38.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.815+0000 7f9def38c700 1 -- 172.21.15.78:0/3007831371 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9de80c4540 msgr2=0x7f9de80c02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:38.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.815+0000 7f9def38c700 1 --2- 172.21.15.78:0/3007831371 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9de80c4540 0x7f9de80c02e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9de40098f0 tx=0x7f9de4009c00 comp rx=0 tx=0).stop 2023-10-14T11:58:38.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.815+0000 7f9def38c700 1 -- 172.21.15.78:0/3007831371 shutdown_connections 2023-10-14T11:58:38.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.815+0000 7f9def38c700 1 --2- 172.21.15.78:0/3007831371 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9de813dbe0 0x7f9de806a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:38.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.815+0000 7f9def38c700 1 --2- 172.21.15.78:0/3007831371 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9de80c4540 0x7f9de80c02e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:38.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.815+0000 7f9def38c700 1 -- 172.21.15.78:0/3007831371 >> 172.21.15.78:0/3007831371 conn(0x7f9de80c3a50 msgr2=0x7f9de80c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:38.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.816+0000 7f9def38c700 1 -- 172.21.15.78:0/3007831371 shutdown_connections 2023-10-14T11:58:38.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.816+0000 7f9def38c700 1 -- 172.21.15.78:0/3007831371 wait complete. 2023-10-14T11:58:38.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.817+0000 7f9def38c700 1 Processor -- start 2023-10-14T11:58:38.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.817+0000 7f9def38c700 1 -- start start 2023-10-14T11:58:38.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.817+0000 7f9def38c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9de806ae10 0x7f9de80c0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:38.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.817+0000 7f9def38c700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9de80c4540 0x7f9de8137a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:38.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.817+0000 7f9def38c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9de813dbe0 0x7f9de8137f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:38.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.817+0000 7f9def38c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de8158610 con 0x7f9de813dbe0 2023-10-14T11:58:38.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.817+0000 7f9def38c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de81ead70 con 0x7f9de806ae10 2023-10-14T11:58:38.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.817+0000 7f9def38c700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9de81588b0 con 0x7f9de80c4540 2023-10-14T11:58:38.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.817+0000 7f9ded929700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9de813dbe0 0x7f9de8137f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:38.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.817+0000 7f9ded929700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9de813dbe0 0x7f9de8137f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45448/0 (socket says 172.21.15.78:45448) 2023-10-14T11:58:38.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.817+0000 7f9ded929700 1 -- 172.21.15.78:0/1415517262 learned_addr learned my addr 172.21.15.78:0/1415517262 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:38.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.817+0000 7f9dec927700 1 --2- 172.21.15.78:0/1415517262 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9de80c4540 0x7f9de8137a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:38.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.818+0000 7f9ded128700 1 --2- 172.21.15.78:0/1415517262 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9de806ae10 0x7f9de80c0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:38.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.818+0000 7f9ded929700 1 -- 172.21.15.78:0/1415517262 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9de80c4540 msgr2=0x7f9de8137a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:38.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.818+0000 7f9ded929700 1 --2- 172.21.15.78:0/1415517262 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9de80c4540 0x7f9de8137a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:38.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.818+0000 7f9ded929700 1 -- 172.21.15.78:0/1415517262 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9de806ae10 msgr2=0x7f9de80c0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:38.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.818+0000 7f9ded929700 1 --2- 172.21.15.78:0/1415517262 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9de806ae10 0x7f9de80c0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:38.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.818+0000 7f9ded929700 1 -- 172.21.15.78:0/1415517262 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de4009550 con 0x7f9de813dbe0 2023-10-14T11:58:38.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.818+0000 7f9ded929700 1 --2- 172.21.15.78:0/1415517262 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9de813dbe0 0x7f9de8137f50 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9de000cb90 tx=0x7f9de000cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:38.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.818+0000 7f9dddffb700 1 -- 172.21.15.78:0/1415517262 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de000aac0 con 0x7f9de813dbe0 2023-10-14T11:58:38.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.818+0000 7f9dddffb700 1 -- 172.21.15.78:0/1415517262 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de000ac40 con 0x7f9de813dbe0 2023-10-14T11:58:38.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.818+0000 7f9def38c700 1 -- 172.21.15.78:0/1415517262 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9de8158b90 con 0x7f9de813dbe0 2023-10-14T11:58:38.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.818+0000 7f9dddffb700 1 -- 172.21.15.78:0/1415517262 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de00070d0 con 0x7f9de813dbe0 2023-10-14T11:58:38.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.818+0000 7f9def38c700 1 -- 172.21.15.78:0/1415517262 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9de80c1030 con 0x7f9de813dbe0 2023-10-14T11:58:38.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.819+0000 7f9def38c700 1 -- 172.21.15.78:0/1415517262 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9de8158d20 con 0x7f9de813dbe0 2023-10-14T11:58:38.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.819+0000 7f9dddffb700 1 -- 172.21.15.78:0/1415517262 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f9de0007250 con 0x7f9de813dbe0 2023-10-14T11:58:38.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.820+0000 7f9dddffb700 1 --2- 172.21.15.78:0/1415517262 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f9dcc06af00 0x7f9dcc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:38.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.820+0000 7f9ded128700 1 --2- 172.21.15.78:0/1415517262 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f9dcc06af00 0x7f9dcc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:38.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.820+0000 7f9dddffb700 1 -- 172.21.15.78:0/1415517262 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9de0089360 con 0x7f9de813dbe0 2023-10-14T11:58:38.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.820+0000 7f9ded128700 1 --2- 172.21.15.78:0/1415517262 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f9dcc06af00 0x7f9dcc06d3c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9dd800a950 tx=0x7f9dd8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:38.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:38.824+0000 7f9dddffb700 1 -- 172.21.15.78:0/1415517262 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9de00544a0 con 0x7f9de813dbe0 2023-10-14T11:58:39.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.014+0000 7f9def38c700 1 -- 172.21.15.78:0/1415517262 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9de8074b80 con 0x7f9de813dbe0 2023-10-14T11:58:39.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.014+0000 7f9dddffb700 1 -- 172.21.15.78:0/1415517262 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9de0018020 con 0x7f9de813dbe0 2023-10-14T11:58:39.017 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2023-10-14T11:58:39.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.017+0000 7f9def38c700 1 -- 172.21.15.78:0/1415517262 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f9dcc06af00 msgr2=0x7f9dcc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:39.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.017+0000 7f9def38c700 1 --2- 172.21.15.78:0/1415517262 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f9dcc06af00 0x7f9dcc06d3c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9dd800a950 tx=0x7f9dd8008040 comp rx=0 tx=0).stop 2023-10-14T11:58:39.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.017+0000 7f9def38c700 1 -- 172.21.15.78:0/1415517262 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9de813dbe0 msgr2=0x7f9de8137f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:39.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.017+0000 7f9def38c700 1 --2- 172.21.15.78:0/1415517262 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9de813dbe0 0x7f9de8137f50 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9de000cb90 tx=0x7f9de000cea0 comp rx=0 tx=0).stop 2023-10-14T11:58:39.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.017+0000 7f9def38c700 1 -- 172.21.15.78:0/1415517262 shutdown_connections 2023-10-14T11:58:39.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.017+0000 7f9def38c700 1 --2- 172.21.15.78:0/1415517262 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9de813dbe0 0x7f9de8137f50 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:39.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.017+0000 7f9def38c700 1 --2- 172.21.15.78:0/1415517262 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f9dcc06af00 0x7f9dcc06d3c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:39.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.017+0000 7f9def38c700 1 --2- 172.21.15.78:0/1415517262 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9de806ae10 0x7f9de80c0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:39.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.017+0000 7f9def38c700 1 --2- 172.21.15.78:0/1415517262 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9de80c4540 0x7f9de8137a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:39.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.017+0000 7f9def38c700 1 -- 172.21.15.78:0/1415517262 >> 172.21.15.78:0/1415517262 conn(0x7f9de80c3a50 msgr2=0x7f9de813ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:39.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.017+0000 7f9def38c700 1 -- 172.21.15.78:0/1415517262 shutdown_connections 2023-10-14T11:58:39.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.017+0000 7f9def38c700 1 -- 172.21.15.78:0/1415517262 wait complete. 2023-10-14T11:58:39.031 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T11:58:39.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.148+0000 7f05777e1700 1 Processor -- start 2023-10-14T11:58:39.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.149+0000 7f05777e1700 1 -- start start 2023-10-14T11:58:39.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.149+0000 7f05777e1700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f05700c44c0 0x7f05700c7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:39.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.149+0000 7f05777e1700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f05700c72a0 0x7f05700c86e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:39.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.149+0000 7f05777e1700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f05700c8c20 0x7f05700c2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:39.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.149+0000 7f05777e1700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0570156b20 con 0x7f05700c8c20 2023-10-14T11:58:39.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.149+0000 7f05777e1700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0570155f10 con 0x7f05700c44c0 2023-10-14T11:58:39.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.149+0000 7f05777e1700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05701561f0 con 0x7f05700c72a0 2023-10-14T11:58:39.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.149+0000 7f0575d7e700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f05700c8c20 0x7f05700c2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:39.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.149+0000 7f0575d7e700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f05700c8c20 0x7f05700c2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45456/0 (socket says 172.21.15.78:45456) 2023-10-14T11:58:39.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.149+0000 7f0575d7e700 1 -- 172.21.15.78:0/4164227997 learned_addr learned my addr 172.21.15.78:0/4164227997 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:39.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.149+0000 7f057557d700 1 --2- 172.21.15.78:0/4164227997 >> v2:172.21.15.195:3300/0 conn(0x7f05700c44c0 0x7f05700c7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:39.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.150+0000 7f0574d7c700 1 --2- 172.21.15.78:0/4164227997 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f05700c72a0 0x7f05700c86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:39.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.150+0000 7f0575d7e700 1 -- 172.21.15.78:0/4164227997 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f05700c72a0 msgr2=0x7f05700c86e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:39.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.150+0000 7f0575d7e700 1 --2- 172.21.15.78:0/4164227997 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f05700c72a0 0x7f05700c86e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:39.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.150+0000 7f0575d7e700 1 -- 172.21.15.78:0/4164227997 >> v2:172.21.15.195:3300/0 conn(0x7f05700c44c0 msgr2=0x7f05700c7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:39.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.150+0000 7f0575d7e700 1 --2- 172.21.15.78:0/4164227997 >> v2:172.21.15.195:3300/0 conn(0x7f05700c44c0 0x7f05700c7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:39.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.150+0000 7f0575d7e700 1 -- 172.21.15.78:0/4164227997 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05701564d0 con 0x7f05700c8c20 2023-10-14T11:58:39.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.150+0000 7f0575d7e700 1 --2- 172.21.15.78:0/4164227997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05700c8c20 0x7f05700c2c90 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f05700c9210 tx=0x7f056800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35587905b181c7c1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:39.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.150+0000 7f0567fff700 1 -- 172.21.15.78:0/4164227997 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056800e070 con 0x7f05700c8c20 2023-10-14T11:58:39.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.150+0000 7f0567fff700 1 -- 172.21.15.78:0/4164227997 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f056800b910 con 0x7f05700c8c20 2023-10-14T11:58:39.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.150+0000 7f0567fff700 1 -- 172.21.15.78:0/4164227997 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056800bcc0 con 0x7f05700c8c20 2023-10-14T11:58:39.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.151+0000 7f05777e1700 1 -- 172.21.15.78:0/4164227997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05700c8c20 msgr2=0x7f05700c2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:39.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.151+0000 7f05777e1700 1 --2- 172.21.15.78:0/4164227997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05700c8c20 0x7f05700c2c90 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f05700c9210 tx=0x7f056800b330 comp rx=0 tx=0).stop 2023-10-14T11:58:39.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.151+0000 7f05777e1700 1 -- 172.21.15.78:0/4164227997 shutdown_connections 2023-10-14T11:58:39.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.151+0000 7f05777e1700 1 --2- 172.21.15.78:0/4164227997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05700c8c20 0x7f05700c2c90 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:39.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.151+0000 7f05777e1700 1 --2- 172.21.15.78:0/4164227997 >> v2:172.21.15.195:3300/0 conn(0x7f05700c44c0 0x7f05700c7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:39.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.151+0000 7f05777e1700 1 --2- 172.21.15.78:0/4164227997 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f05700c72a0 0x7f05700c86e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:39.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.151+0000 7f05777e1700 1 -- 172.21.15.78:0/4164227997 >> 172.21.15.78:0/4164227997 conn(0x7f05700737d0 msgr2=0x7f0570073be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:39.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.151+0000 7f05777e1700 1 -- 172.21.15.78:0/4164227997 shutdown_connections 2023-10-14T11:58:39.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.151+0000 7f05777e1700 1 -- 172.21.15.78:0/4164227997 wait complete. 2023-10-14T11:58:39.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.152+0000 7f05777e1700 1 Processor -- start 2023-10-14T11:58:39.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.152+0000 7f05777e1700 1 -- start start 2023-10-14T11:58:39.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.153+0000 7f05777e1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05700c44c0 0x7f05700bde40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:39.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.153+0000 7f05777e1700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f05700c72a0 0x7f05700ad610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:39.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.153+0000 7f05777e1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f05700c8c20 0x7f05700c1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:39.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.153+0000 7f05777e1700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05700bc9b0 con 0x7f05700c44c0 2023-10-14T11:58:39.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.153+0000 7f05777e1700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05701e8830 con 0x7f05700c8c20 2023-10-14T11:58:39.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.153+0000 7f05777e1700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05700bc3f0 con 0x7f05700c72a0 2023-10-14T11:58:39.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.153+0000 7f057557d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05700c44c0 0x7f05700bde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:39.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.153+0000 7f057557d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05700c44c0 0x7f05700bde40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45458/0 (socket says 172.21.15.78:45458) 2023-10-14T11:58:39.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.153+0000 7f057557d700 1 -- 172.21.15.78:0/214634327 learned_addr learned my addr 172.21.15.78:0/214634327 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:39.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.153+0000 7f0574d7c700 1 --2- 172.21.15.78:0/214634327 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f05700c72a0 0x7f05700ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:39.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.153+0000 7f0575d7e700 1 --2- 172.21.15.78:0/214634327 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f05700c8c20 0x7f05700c1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:39.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.153+0000 7f057557d700 1 -- 172.21.15.78:0/214634327 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f05700c72a0 msgr2=0x7f05700ad610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:39.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.153+0000 7f057557d700 1 --2- 172.21.15.78:0/214634327 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f05700c72a0 0x7f05700ad610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:39.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.153+0000 7f057557d700 1 -- 172.21.15.78:0/214634327 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f05700c8c20 msgr2=0x7f05700c1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:39.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.153+0000 7f057557d700 1 --2- 172.21.15.78:0/214634327 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f05700c8c20 0x7f05700c1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:39.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.153+0000 7f057557d700 1 -- 172.21.15.78:0/214634327 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0568009b30 con 0x7f05700c44c0 2023-10-14T11:58:39.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.153+0000 7f0575d7e700 1 --2- 172.21.15.78:0/214634327 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f05700c8c20 0x7f05700c1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:39.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.153+0000 7f057557d700 1 --2- 172.21.15.78:0/214634327 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05700c44c0 0x7f05700bde40 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f056000e930 tx=0x7f056000ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:39.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.153+0000 7f0565ffb700 1 -- 172.21.15.78:0/214634327 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056000ee80 con 0x7f05700c44c0 2023-10-14T11:58:39.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.153+0000 7f0565ffb700 1 -- 172.21.15.78:0/214634327 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f056000c850 con 0x7f05700c44c0 2023-10-14T11:58:39.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.153+0000 7f0565ffb700 1 -- 172.21.15.78:0/214634327 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056000cbe0 con 0x7f05700c44c0 2023-10-14T11:58:39.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.153+0000 7f05777e1700 1 -- 172.21.15.78:0/214634327 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05700bc6d0 con 0x7f05700c44c0 2023-10-14T11:58:39.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.154+0000 7f05777e1700 1 -- 172.21.15.78:0/214634327 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05700adb50 con 0x7f05700c44c0 2023-10-14T11:58:39.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.155+0000 7f05777e1700 1 -- 172.21.15.78:0/214634327 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05700bbbb0 con 0x7f05700c44c0 2023-10-14T11:58:39.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.155+0000 7f0565ffb700 1 -- 172.21.15.78:0/214634327 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f056000cd40 con 0x7f05700c44c0 2023-10-14T11:58:39.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.156+0000 7f0565ffb700 1 --2- 172.21.15.78:0/214634327 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f055406af00 0x7f055406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:39.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.156+0000 7f0565ffb700 1 -- 172.21.15.78:0/214634327 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0560089670 con 0x7f05700c44c0 2023-10-14T11:58:39.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.156+0000 7f0574d7c700 1 --2- 172.21.15.78:0/214634327 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f055406af00 0x7f055406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:39.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.156+0000 7f0574d7c700 1 --2- 172.21.15.78:0/214634327 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f055406af00 0x7f055406d3c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f05700c8870 tx=0x7f056c009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:39.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.160+0000 7f0565ffb700 1 -- 172.21.15.78:0/214634327 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f05700bbbb0 con 0x7f05700c44c0 2023-10-14T11:58:39.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.353+0000 7f05777e1700 1 -- 172.21.15.78:0/214634327 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f05700bbbb0 con 0x7f05700c44c0 2023-10-14T11:58:39.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.354+0000 7f0565ffb700 1 -- 172.21.15.78:0/214634327 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16481 (secure 0 0 0) 0x7f05700bbbb0 con 0x7f05700c44c0 2023-10-14T11:58:39.357 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:39.357 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":21,"fsid":"4c5dc134-3092-406a-a1e1-13e6d9e926e8","created":"2023-10-14T11:57:57.363750+0000","modified":"2023-10-14T11:58:16.139413+0000","last_up_change":"2023-10-14T11:58:06.982472+0000","last_in_change":"2023-10-14T11:57:59.821373+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T11:58:09.001200+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T11:58:12.778573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T11:58:13.490286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"625a5fc0-9474-4eae-b80e-14c27a74ad19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6801","nonce":3884969780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6803","nonce":3884969780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6807","nonce":3884969780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6805","nonce":3884969780}]},"public_addr":"172.21.15.78:6801/3884969780","cluster_addr":"172.21.15.78:6803/3884969780","heartbeat_back_addr":"172.21.15.78:6807/3884969780","heartbeat_front_addr":"172.21.15.78:6805/3884969780","state":["exists","up"]},{"osd":1,"uuid":"49d63db1-2981-49ab-9c40-c50fc744b982","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6809","nonce":2176928406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6811","nonce":2176928406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6815","nonce":2176928406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6813","nonce":2176928406}]},"public_addr":"172.21.15.78:6809/2176928406","cluster_addr":"172.21.15.78:6811/2176928406","heartbeat_back_addr":"172.21.15.78:6815/2176928406","heartbeat_front_addr":"172.21.15.78:6813/2176928406","state":["exists","up"]},{"osd":2,"uuid":"1d7d0b29-cbd1-4717-9fe6-765a8d4998b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6825","nonce":3647044577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6827","nonce":3647044577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6831","nonce":3647044577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6829","nonce":3647044577}]},"public_addr":"172.21.15.78:6825/3647044577","cluster_addr":"172.21.15.78:6827/3647044577","heartbeat_back_addr":"172.21.15.78:6831/3647044577","heartbeat_front_addr":"172.21.15.78:6829/3647044577","state":["exists","up"]},{"osd":3,"uuid":"2f75ded1-e9e2-40e7-99ae-45e1e425e1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6817","nonce":3584112288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6819","nonce":3584112288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6823","nonce":3584112288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6821","nonce":3584112288}]},"public_addr":"172.21.15.78:6817/3584112288","cluster_addr":"172.21.15.78:6819/3584112288","heartbeat_back_addr":"172.21.15.78:6823/3584112288","heartbeat_front_addr":"172.21.15.78:6821/3584112288","state":["exists","up"]},{"osd":4,"uuid":"5130b432-8358-46cb-8a31-3064c976f6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6801","nonce":3732756301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6803","nonce":3732756301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6807","nonce":3732756301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6805","nonce":3732756301}]},"public_addr":"172.21.15.195:6801/3732756301","cluster_addr":"172.21.15.195:6803/3732756301","heartbeat_back_addr":"172.21.15.195:6807/3732756301","heartbeat_front_addr":"172.21.15.195:6805/3732756301","state":["exists","up"]},{"osd":5,"uuid":"60e234a9-a506-4e40-9c19-49a58f3cf28c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6825","nonce":2857706411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6827","nonce":2857706411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6831","nonce":2857706411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6829","nonce":2857706411}]},"public_addr":"172.21.15.195:6825/2857706411","cluster_addr":"172.21.15.195:6827/2857706411","heartbeat_back_addr":"172.21.15.195:6831/2857706411","heartbeat_front_addr":"172.21.15.195:6829/2857706411","state":["exists","up"]},{"osd":6,"uuid":"df7d2dd6-26b6-4eb1-96e5-94d1f3955b33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6810","nonce":72554374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6813","nonce":72554374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6820","nonce":72554374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6817","nonce":72554374}]},"public_addr":"172.21.15.195:6810/72554374","cluster_addr":"172.21.15.195:6813/72554374","heartbeat_back_addr":"172.21.15.195:6820/72554374","heartbeat_front_addr":"172.21.15.195:6817/72554374","state":["exists","up"]},{"osd":7,"uuid":"47a9d4de-f56d-4aa4-9900-c4a9cf9b2f97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6809","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6811","nonce":482472979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6816","nonce":482472979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6823","nonce":482472979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6821","nonce":482472979}]},"public_addr":"172.21.15.195:6811/482472979","cluster_addr":"172.21.15.195:6816/482472979","heartbeat_back_addr":"172.21.15.195:6823/482472979","heartbeat_front_addr":"172.21.15.195:6821/482472979","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.754003+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.005994+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.043603+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.024506+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.062585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.033767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.994825+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.038300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T11:58:39.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.357+0000 7f05777e1700 1 -- 172.21.15.78:0/214634327 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f055406af00 msgr2=0x7f055406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:39.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.357+0000 7f05777e1700 1 --2- 172.21.15.78:0/214634327 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f055406af00 0x7f055406d3c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f05700c8870 tx=0x7f056c009210 comp rx=0 tx=0).stop 2023-10-14T11:58:39.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.357+0000 7f05777e1700 1 -- 172.21.15.78:0/214634327 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05700c44c0 msgr2=0x7f05700bde40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:39.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.357+0000 7f05777e1700 1 --2- 172.21.15.78:0/214634327 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05700c44c0 0x7f05700bde40 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f056000e930 tx=0x7f056000ec40 comp rx=0 tx=0).stop 2023-10-14T11:58:39.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.357+0000 7f05777e1700 1 -- 172.21.15.78:0/214634327 shutdown_connections 2023-10-14T11:58:39.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.357+0000 7f05777e1700 1 --2- 172.21.15.78:0/214634327 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f055406af00 0x7f055406d3c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:39.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.357+0000 7f05777e1700 1 --2- 172.21.15.78:0/214634327 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f05700c8c20 0x7f05700c1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:39.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.357+0000 7f05777e1700 1 --2- 172.21.15.78:0/214634327 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05700c44c0 0x7f05700bde40 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:39.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.357+0000 7f05777e1700 1 --2- 172.21.15.78:0/214634327 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f05700c72a0 0x7f05700ad610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:39.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.357+0000 7f05777e1700 1 -- 172.21.15.78:0/214634327 >> 172.21.15.78:0/214634327 conn(0x7f05700737d0 msgr2=0x7f05700a9140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:39.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.357+0000 7f05777e1700 1 -- 172.21.15.78:0/214634327 shutdown_connections 2023-10-14T11:58:39.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.357+0000 7f05777e1700 1 -- 172.21.15.78:0/214634327 wait complete. 2023-10-14T11:58:39.371 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T11:58:09.001200+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T11:58:12.778573+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T11:58:13.490286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T11:58:39.371 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T11:58:39.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.492+0000 7fa4027a1700 1 Processor -- start 2023-10-14T11:58:39.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.492+0000 7fa4027a1700 1 -- start start 2023-10-14T11:58:39.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.493+0000 7fa4027a1700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa3fc1533d0 0x7fa3fc0c8680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:39.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.493+0000 7fa4027a1700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3fc1537b0 con 0x7fa3fc0a70c0 2023-10-14T11:58:39.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.493+0000 7fa4027a1700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3fc0c7c50 con 0x7fa3fc0c8bc0 2023-10-14T11:58:39.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.493+0000 7fa4027a1700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3fc0c6fb0 con 0x7fa3fc1533d0 2023-10-14T11:58:39.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.493+0000 7fa3fbfff700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa3fc1533d0 0x7fa3fc0c8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:39.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.493+0000 7fa3fbfff700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa3fc1533d0 0x7fa3fc0c8680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45466/0 (socket says 172.21.15.78:45466) 2023-10-14T11:58:39.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.493+0000 7fa400d3e700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fa3fc0a70c0 0x7fa3fc155070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50346/0 (socket says 172.21.15.78:50346) 2023-10-14T11:58:39.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.493+0000 7fa3fbfff700 1 -- 172.21.15.78:0/3101304985 learned_addr learned my addr 172.21.15.78:0/3101304985 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:39.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.493+0000 7fa3fbfff700 1 -- 172.21.15.78:0/3101304985 >> v1:172.21.15.195:6789/0 conn(0x7fa3fc0c8bc0 legacy=0x7fa3fc154960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:39.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.493+0000 7fa3fbfff700 1 -- 172.21.15.78:0/3101304985 >> v1:172.21.15.78:6789/0 conn(0x7fa3fc0a70c0 legacy=0x7fa3fc155070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:39.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.493+0000 7fa3fbfff700 1 -- 172.21.15.78:0/3101304985 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3fc153e80 con 0x7fa3fc1533d0 2023-10-14T11:58:39.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.493+0000 7fa3faffd700 1 -- 172.21.15.78:0/3101304985 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3759287319 0 0) 0x7fa3fc1537b0 con 0x7fa3fc0a70c0 2023-10-14T11:58:39.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.493+0000 7fa3fbfff700 1 --2- 172.21.15.78:0/3101304985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3fc1533d0 0x7fa3fc0c8680 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa3e40098f0 tx=0x7fa3e4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95ac4abf6e0ffeed server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:39.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.494+0000 7fa3faffd700 1 -- 172.21.15.78:0/3101304985 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e4004230 con 0x7fa3fc1533d0 2023-10-14T11:58:39.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.494+0000 7fa3faffd700 1 -- 172.21.15.78:0/3101304985 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3e4004390 con 0x7fa3fc1533d0 2023-10-14T11:58:39.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.494+0000 7fa3faffd700 1 -- 172.21.15.78:0/3101304985 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e40036a0 con 0x7fa3fc1533d0 2023-10-14T11:58:39.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.494+0000 7fa4027a1700 1 -- 172.21.15.78:0/3101304985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3fc1533d0 msgr2=0x7fa3fc0c8680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:39.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.494+0000 7fa4027a1700 1 --2- 172.21.15.78:0/3101304985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3fc1533d0 0x7fa3fc0c8680 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa3e40098f0 tx=0x7fa3e4009c00 comp rx=0 tx=0).stop 2023-10-14T11:58:39.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.494+0000 7fa4027a1700 1 -- 172.21.15.78:0/3101304985 shutdown_connections 2023-10-14T11:58:39.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.494+0000 7fa4027a1700 1 --2- 172.21.15.78:0/3101304985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3fc1533d0 0x7fa3fc0c8680 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:39.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.494+0000 7fa4027a1700 1 -- 172.21.15.78:0/3101304985 >> 172.21.15.78:0/3101304985 conn(0x7fa3fc0737d0 msgr2=0x7fa3fc073be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:39.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.494+0000 7fa4027a1700 1 -- 172.21.15.78:0/3101304985 shutdown_connections 2023-10-14T11:58:39.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.494+0000 7fa4027a1700 1 -- 172.21.15.78:0/3101304985 wait complete. 2023-10-14T11:58:39.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.495+0000 7fa4027a1700 1 Processor -- start 2023-10-14T11:58:39.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.496+0000 7fa4027a1700 1 -- start start 2023-10-14T11:58:39.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.496+0000 7fa4027a1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa3fc0a70c0 0x7fa3fc0be790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:39.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.496+0000 7fa4027a1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3fc0c8bc0 0x7fa3fc1e8560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:39.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.496+0000 7fa4027a1700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa3fc1533d0 0x7fa3fc0ad720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:39.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.496+0000 7fa4027a1700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc0c7f30 con 0x7fa3fc0c8bc0 2023-10-14T11:58:39.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.496+0000 7fa4027a1700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc0c7a70 con 0x7fa3fc0a70c0 2023-10-14T11:58:39.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.496+0000 7fa4027a1700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc0c19d0 con 0x7fa3fc1533d0 2023-10-14T11:58:39.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.496+0000 7fa3fb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3fc0c8bc0 0x7fa3fc1e8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:39.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.496+0000 7fa3fb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3fc0c8bc0 0x7fa3fc1e8560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45474/0 (socket says 172.21.15.78:45474) 2023-10-14T11:58:39.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.496+0000 7fa3fb7fe700 1 -- 172.21.15.78:0/3946029506 learned_addr learned my addr 172.21.15.78:0/3946029506 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:39.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.496+0000 7fa400d3e700 1 --2- 172.21.15.78:0/3946029506 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa3fc1533d0 0x7fa3fc0ad720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:39.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.497+0000 7fa3fb7fe700 1 -- 172.21.15.78:0/3946029506 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa3fc1533d0 msgr2=0x7fa3fc0ad720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:39.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.497+0000 7fa3fbfff700 1 --2- 172.21.15.78:0/3946029506 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa3fc0a70c0 0x7fa3fc0be790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:39.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.497+0000 7fa3fb7fe700 1 --2- 172.21.15.78:0/3946029506 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa3fc1533d0 0x7fa3fc0ad720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:39.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.497+0000 7fa3fb7fe700 1 -- 172.21.15.78:0/3946029506 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa3fc0a70c0 msgr2=0x7fa3fc0be790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:39.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.497+0000 7fa3fb7fe700 1 --2- 172.21.15.78:0/3946029506 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa3fc0a70c0 0x7fa3fc0be790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:39.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.497+0000 7fa3fb7fe700 1 -- 172.21.15.78:0/3946029506 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e4009550 con 0x7fa3fc0c8bc0 2023-10-14T11:58:39.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.497+0000 7fa400d3e700 1 --2- 172.21.15.78:0/3946029506 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa3fc1533d0 0x7fa3fc0ad720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:39.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.497+0000 7fa3fb7fe700 1 --2- 172.21.15.78:0/3946029506 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3fc0c8bc0 0x7fa3fc1e8560 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec00c4d0 tx=0x7fa3ec00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:39.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.497+0000 7fa3f8ff9700 1 -- 172.21.15.78:0/3946029506 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec0054e0 con 0x7fa3fc0c8bc0 2023-10-14T11:58:39.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.497+0000 7fa3f8ff9700 1 -- 172.21.15.78:0/3946029506 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3ec003680 con 0x7fa3fc0c8bc0 2023-10-14T11:58:39.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.497+0000 7fa3f8ff9700 1 -- 172.21.15.78:0/3946029506 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec003a30 con 0x7fa3fc0c8bc0 2023-10-14T11:58:39.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.497+0000 7fa4027a1700 1 -- 172.21.15.78:0/3946029506 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3fc0becd0 con 0x7fa3fc0c8bc0 2023-10-14T11:58:39.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.497+0000 7fa4027a1700 1 -- 172.21.15.78:0/3946029506 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3fc1e8aa0 con 0x7fa3fc0c8bc0 2023-10-14T11:58:39.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.498+0000 7fa4027a1700 1 -- 172.21.15.78:0/3946029506 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3fc0c1b10 con 0x7fa3fc0c8bc0 2023-10-14T11:58:39.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.498+0000 7fa3f8ff9700 1 -- 172.21.15.78:0/3946029506 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa3ec003b90 con 0x7fa3fc0c8bc0 2023-10-14T11:58:39.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.499+0000 7fa3f8ff9700 1 --2- 172.21.15.78:0/3946029506 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa3dc06aed0 0x7fa3dc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:39.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.499+0000 7fa3f8ff9700 1 -- 172.21.15.78:0/3946029506 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa3ec088470 con 0x7fa3fc0c8bc0 2023-10-14T11:58:39.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.499+0000 7fa3fbfff700 1 --2- 172.21.15.78:0/3946029506 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa3dc06aed0 0x7fa3dc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:39.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.499+0000 7fa3fbfff700 1 --2- 172.21.15.78:0/3946029506 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa3dc06aed0 0x7fa3dc06d390 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa3e4009c40 tx=0x7fa3e4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:39.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.503+0000 7fa3f8ff9700 1 -- 172.21.15.78:0/3946029506 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa3fc0c1b10 con 0x7fa3fc0c8bc0 2023-10-14T11:58:39.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.692+0000 7fa4027a1700 1 -- 172.21.15.78:0/3946029506 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa3fc1e8aa0 con 0x7fa3fc0c8bc0 2023-10-14T11:58:39.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.693+0000 7fa3f8ff9700 1 -- 172.21.15.78:0/3946029506 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa3fc1e8aa0 con 0x7fa3fc0c8bc0 2023-10-14T11:58:39.695 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-10-14T11:58:39.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.695+0000 7fa4027a1700 1 -- 172.21.15.78:0/3946029506 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa3dc06aed0 msgr2=0x7fa3dc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:39.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.695+0000 7fa4027a1700 1 --2- 172.21.15.78:0/3946029506 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa3dc06aed0 0x7fa3dc06d390 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa3e4009c40 tx=0x7fa3e4008030 comp rx=0 tx=0).stop 2023-10-14T11:58:39.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.695+0000 7fa4027a1700 1 -- 172.21.15.78:0/3946029506 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3fc0c8bc0 msgr2=0x7fa3fc1e8560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:39.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.695+0000 7fa4027a1700 1 --2- 172.21.15.78:0/3946029506 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3fc0c8bc0 0x7fa3fc1e8560 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec00c4d0 tx=0x7fa3ec00c890 comp rx=0 tx=0).stop 2023-10-14T11:58:39.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.695+0000 7fa4027a1700 1 -- 172.21.15.78:0/3946029506 shutdown_connections 2023-10-14T11:58:39.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.695+0000 7fa4027a1700 1 --2- 172.21.15.78:0/3946029506 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa3fc1533d0 0x7fa3fc0ad720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:39.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.695+0000 7fa4027a1700 1 --2- 172.21.15.78:0/3946029506 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa3dc06aed0 0x7fa3dc06d390 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:39.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.695+0000 7fa4027a1700 1 --2- 172.21.15.78:0/3946029506 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa3fc0a70c0 0x7fa3fc0be790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:39.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.695+0000 7fa4027a1700 1 --2- 172.21.15.78:0/3946029506 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3fc0c8bc0 0x7fa3fc1e8560 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:39.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.695+0000 7fa4027a1700 1 -- 172.21.15.78:0/3946029506 >> 172.21.15.78:0/3946029506 conn(0x7fa3fc0737d0 msgr2=0x7fa3fc1525d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:39.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.695+0000 7fa4027a1700 1 -- 172.21.15.78:0/3946029506 shutdown_connections 2023-10-14T11:58:39.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.695+0000 7fa4027a1700 1 -- 172.21.15.78:0/3946029506 wait complete. 2023-10-14T11:58:39.709 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T11:58:39.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.829+0000 7f7ef61cb700 1 Processor -- start 2023-10-14T11:58:39.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.829+0000 7f7ef61cb700 1 -- start start 2023-10-14T11:58:39.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.830+0000 7f7ef61cb700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7ef0155520 0x7f7ef0156b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:39.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.830+0000 7f7ef61cb700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ef0155f90 con 0x7f7ef0157900 2023-10-14T11:58:39.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.830+0000 7f7ef61cb700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ef0071f70 con 0x7f7ef006dee0 2023-10-14T11:58:39.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.830+0000 7f7ef61cb700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef006db60 con 0x7f7ef0155520 2023-10-14T11:58:39.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.830+0000 7f7eef7fe700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f7ef0157900 0x7f7ef00c6770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50356/0 (socket says 172.21.15.78:50356) 2023-10-14T11:58:39.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.830+0000 7f7eef7fe700 1 -- 172.21.15.78:0/4151419528 learned_addr learned my addr 172.21.15.78:0/4151419528 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:39.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.830+0000 7f7eeffff700 1 --2- 172.21.15.78:0/4151419528 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7ef0155520 0x7f7ef0156b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:39.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.830+0000 7f7eee7fc700 1 -- 172.21.15.78:0/4151419528 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3464384165 0 0) 0x7f7ef0155f90 con 0x7f7ef0157900 2023-10-14T11:58:39.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.830+0000 7f7eee7fc700 1 -- 172.21.15.78:0/4151419528 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ed80035b0 con 0x7f7ef0157900 2023-10-14T11:58:39.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.831+0000 7f7eee7fc700 1 -- 172.21.15.78:0/4151419528 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 256176450 0 0) 0x7f7ed80035b0 con 0x7f7ef0157900 2023-10-14T11:58:39.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.831+0000 7f7eee7fc700 1 -- 172.21.15.78:0/4151419528 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7ef0155520 msgr2=0x7f7ef0156b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:39.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.831+0000 7f7eee7fc700 1 --2- 172.21.15.78:0/4151419528 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7ef0155520 0x7f7ef0156b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:39.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.831+0000 7f7eee7fc700 1 -- 172.21.15.78:0/4151419528 >> v1:172.21.15.195:6789/0 conn(0x7f7ef006dee0 legacy=0x7f7ef0156420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:39.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.831+0000 7f7eee7fc700 1 -- 172.21.15.78:0/4151419528 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef0072250 con 0x7f7ef0157900 2023-10-14T11:58:39.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.831+0000 7f7eee7fc700 1 -- 172.21.15.78:0/4151419528 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441799917 0 0) 0x7f7ee4001670 con 0x7f7ef0157900 2023-10-14T11:58:39.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.831+0000 7f7eee7fc700 1 -- 172.21.15.78:0/4151419528 >> v1:172.21.15.78:6789/0 conn(0x7f7ef0157900 legacy=0x7f7ef00c6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:39.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.831+0000 7f7eee7fc700 1 --2- 172.21.15.78:0/4151419528 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ed8003a30 0x7f7ed8003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:39.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.831+0000 7f7eee7fc700 1 --2- 172.21.15.78:0/4151419528 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7ef0155520 0x7f7ed8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:39.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.831+0000 7f7eee7fc700 1 --2- 172.21.15.78:0/4151419528 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7ed8004d90 0x7f7ed8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:39.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.831+0000 7f7eee7fc700 1 -- 172.21.15.78:0/4151419528 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef006db60 con 0x7f7ed8003a30 2023-10-14T11:58:39.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.831+0000 7f7eee7fc700 1 -- 172.21.15.78:0/4151419528 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed800bbd0 con 0x7f7ed8004d90 2023-10-14T11:58:39.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.831+0000 7f7eee7fc700 1 -- 172.21.15.78:0/4151419528 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed800bd90 con 0x7f7ef0155520 2023-10-14T11:58:39.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.831+0000 7f7eef7fe700 1 --2- 172.21.15.78:0/4151419528 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ed8003a30 0x7f7ed8003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:39.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.831+0000 7f7eeeffd700 1 --2- 172.21.15.78:0/4151419528 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7ef0155520 0x7f7ed8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:39.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.831+0000 7f7eeffff700 1 --2- 172.21.15.78:0/4151419528 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7ed8004d90 0x7f7ed8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:39.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.831+0000 7f7eef7fe700 1 -- 172.21.15.78:0/4151419528 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7ef0155520 msgr2=0x7f7ed8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:39.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.831+0000 7f7eef7fe700 1 --2- 172.21.15.78:0/4151419528 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7ef0155520 0x7f7ed8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:39.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.831+0000 7f7eef7fe700 1 -- 172.21.15.78:0/4151419528 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7ed8004d90 msgr2=0x7f7ed8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:39.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.831+0000 7f7eef7fe700 1 --2- 172.21.15.78:0/4151419528 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7ed8004d90 0x7f7ed8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:39.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.831+0000 7f7eef7fe700 1 -- 172.21.15.78:0/4151419528 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed80035b0 con 0x7f7ed8003a30 2023-10-14T11:58:39.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.831+0000 7f7eeeffd700 1 --2- 172.21.15.78:0/4151419528 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7ef0155520 0x7f7ed8004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:39.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.831+0000 7f7eef7fe700 1 --2- 172.21.15.78:0/4151419528 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ed8003a30 0x7f7ed8003e50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7ee400c240 tx=0x7f7ed8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f030e33af26ca891 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:39.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.831+0000 7f7eee7fc700 1 -- 172.21.15.78:0/4151419528 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee4007c10 con 0x7f7ed8003a30 2023-10-14T11:58:39.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.831+0000 7f7eee7fc700 1 -- 172.21.15.78:0/4151419528 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ee400cde0 con 0x7f7ed8003a30 2023-10-14T11:58:39.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.831+0000 7f7eee7fc700 1 -- 172.21.15.78:0/4151419528 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee4005ba0 con 0x7f7ed8003a30 2023-10-14T11:58:39.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.832+0000 7f7ef61cb700 1 -- 172.21.15.78:0/4151419528 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ed8003a30 msgr2=0x7f7ed8003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:39.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.832+0000 7f7ef61cb700 1 --2- 172.21.15.78:0/4151419528 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ed8003a30 0x7f7ed8003e50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7ee400c240 tx=0x7f7ed8005880 comp rx=0 tx=0).stop 2023-10-14T11:58:39.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.832+0000 7f7eeffff700 1 -- 172.21.15.78:0/4151419528 reap_dead start 2023-10-14T11:58:39.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.832+0000 7f7ef61cb700 1 -- 172.21.15.78:0/4151419528 shutdown_connections 2023-10-14T11:58:39.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.832+0000 7f7ef61cb700 1 -- 172.21.15.78:0/4151419528 >> 172.21.15.78:0/4151419528 conn(0x7f7ef00a8410 msgr2=0x7f7ef00a8820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:39.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.832+0000 7f7ef61cb700 1 -- 172.21.15.78:0/4151419528 shutdown_connections 2023-10-14T11:58:39.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.832+0000 7f7ef61cb700 1 -- 172.21.15.78:0/4151419528 wait complete. 2023-10-14T11:58:39.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.833+0000 7f7ef61cb700 1 Processor -- start 2023-10-14T11:58:39.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.833+0000 7f7ef61cb700 1 -- start start 2023-10-14T11:58:39.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.834+0000 7f7ef61cb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ed8003a30 0x7f7ef00bc880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:39.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.834+0000 7f7ef61cb700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7ef00a7c10 0x7f7ef01ed2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:39.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.834+0000 7f7ef61cb700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7ef00c2350 0x7f7ef01ed800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:39.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.834+0000 7f7ef61cb700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef00c1250 con 0x7f7ed8003a30 2023-10-14T11:58:39.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.834+0000 7f7ef61cb700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef00bc3f0 con 0x7f7ef00c2350 2023-10-14T11:58:39.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.834+0000 7f7ef61cb700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef00c6020 con 0x7f7ef00a7c10 2023-10-14T11:58:39.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.834+0000 7f7eef7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ed8003a30 0x7f7ef00bc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:39.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.834+0000 7f7eef7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ed8003a30 0x7f7ef00bc880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45492/0 (socket says 172.21.15.78:45492) 2023-10-14T11:58:39.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.834+0000 7f7eef7fe700 1 -- 172.21.15.78:0/1620011843 learned_addr learned my addr 172.21.15.78:0/1620011843 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:39.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.834+0000 7f7eeeffd700 1 --2- 172.21.15.78:0/1620011843 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7ef00a7c10 0x7f7ef01ed2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:39.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.834+0000 7f7eef7fe700 1 -- 172.21.15.78:0/1620011843 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7ef00a7c10 msgr2=0x7f7ef01ed2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:39.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.834+0000 7f7eef7fe700 1 --2- 172.21.15.78:0/1620011843 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7ef00a7c10 0x7f7ef01ed2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:39.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.834+0000 7f7eef7fe700 1 -- 172.21.15.78:0/1620011843 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7ef00c2350 msgr2=0x7f7ef01ed800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:39.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.834+0000 7f7eef7fe700 1 --2- 172.21.15.78:0/1620011843 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7ef00c2350 0x7f7ef01ed800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:39.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.834+0000 7f7eef7fe700 1 -- 172.21.15.78:0/1620011843 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef00bda70 con 0x7f7ed8003a30 2023-10-14T11:58:39.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.835+0000 7f7eef7fe700 1 --2- 172.21.15.78:0/1620011843 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ed8003a30 0x7f7ef00bc880 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7ee400c830 tx=0x7f7ee4001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:39.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.835+0000 7f7ed7fff700 1 -- 172.21.15.78:0/1620011843 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee4004a80 con 0x7f7ed8003a30 2023-10-14T11:58:39.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.835+0000 7f7ed7fff700 1 -- 172.21.15.78:0/1620011843 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ee4004be0 con 0x7f7ed8003a30 2023-10-14T11:58:39.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.835+0000 7f7ef61cb700 1 -- 172.21.15.78:0/1620011843 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ef00bdd50 con 0x7f7ed8003a30 2023-10-14T11:58:39.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.835+0000 7f7ed7fff700 1 -- 172.21.15.78:0/1620011843 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee40070d0 con 0x7f7ed8003a30 2023-10-14T11:58:39.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.835+0000 7f7ef61cb700 1 -- 172.21.15.78:0/1620011843 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ef014ebd0 con 0x7f7ed8003a30 2023-10-14T11:58:39.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.836+0000 7f7ef61cb700 1 -- 172.21.15.78:0/1620011843 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ef014ed90 con 0x7f7ed8003a30 2023-10-14T11:58:39.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.836+0000 7f7ed7fff700 1 -- 172.21.15.78:0/1620011843 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7ee4019070 con 0x7f7ed8003a30 2023-10-14T11:58:39.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.836+0000 7f7ed7fff700 1 --2- 172.21.15.78:0/1620011843 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7edc06af30 0x7f7edc06d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:39.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.836+0000 7f7ed7fff700 1 -- 172.21.15.78:0/1620011843 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7ee408a4c0 con 0x7f7ed8003a30 2023-10-14T11:58:39.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.836+0000 7f7eeeffd700 1 --2- 172.21.15.78:0/1620011843 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7edc06af30 0x7f7edc06d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:39.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.837+0000 7f7eeeffd700 1 --2- 172.21.15.78:0/1620011843 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7edc06af30 0x7f7edc06d3f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7ef00a8010 tx=0x7f7ee0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:39.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:39.841+0000 7f7ed7fff700 1 -- 172.21.15.78:0/1620011843 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7ee4055600 con 0x7f7ed8003a30 2023-10-14T11:58:40.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.029+0000 7f7ef61cb700 1 -- 172.21.15.78:0/1620011843 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7ef0074b80 con 0x7f7ed8003a30 2023-10-14T11:58:40.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.030+0000 7f7ed7fff700 1 -- 172.21.15.78:0/1620011843 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7ee4058c20 con 0x7f7ed8003a30 2023-10-14T11:58:40.032 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-10-14T11:58:40.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.032+0000 7f7ef61cb700 1 -- 172.21.15.78:0/1620011843 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7edc06af30 msgr2=0x7f7edc06d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:40.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.032+0000 7f7ef61cb700 1 --2- 172.21.15.78:0/1620011843 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7edc06af30 0x7f7edc06d3f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7ef00a8010 tx=0x7f7ee0009040 comp rx=0 tx=0).stop 2023-10-14T11:58:40.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.032+0000 7f7ef61cb700 1 -- 172.21.15.78:0/1620011843 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ed8003a30 msgr2=0x7f7ef00bc880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:40.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.032+0000 7f7ef61cb700 1 --2- 172.21.15.78:0/1620011843 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ed8003a30 0x7f7ef00bc880 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7ee400c830 tx=0x7f7ee4001a00 comp rx=0 tx=0).stop 2023-10-14T11:58:40.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.032+0000 7f7ef61cb700 1 -- 172.21.15.78:0/1620011843 shutdown_connections 2023-10-14T11:58:40.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.032+0000 7f7ef61cb700 1 --2- 172.21.15.78:0/1620011843 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7edc06af30 0x7f7edc06d3f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:40.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.032+0000 7f7ef61cb700 1 --2- 172.21.15.78:0/1620011843 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7ef00c2350 0x7f7ef01ed800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:40.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.032+0000 7f7ef61cb700 1 --2- 172.21.15.78:0/1620011843 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ed8003a30 0x7f7ef00bc880 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:40.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.032+0000 7f7ef61cb700 1 --2- 172.21.15.78:0/1620011843 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7ef00a7c10 0x7f7ef01ed2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:40.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.032+0000 7f7ef61cb700 1 -- 172.21.15.78:0/1620011843 >> 172.21.15.78:0/1620011843 conn(0x7f7ef00a8410 msgr2=0x7f7ef0157ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:40.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.032+0000 7f7ef61cb700 1 -- 172.21.15.78:0/1620011843 shutdown_connections 2023-10-14T11:58:40.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.032+0000 7f7ef61cb700 1 -- 172.21.15.78:0/1620011843 wait complete. 2023-10-14T11:58:40.046 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T11:58:40.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.167+0000 7fd8d4113700 1 Processor -- start 2023-10-14T11:58:40.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.167+0000 7fd8d4113700 1 -- start start 2023-10-14T11:58:40.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.168+0000 7fd8d4113700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd8cc1533d0 0x7fd8cc0c8680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:40.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.168+0000 7fd8d4113700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd8cc0c8bc0 0x7fd8cc154960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:40.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.168+0000 7fd8d4113700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8cc1537b0 con 0x7fd8cc0a70c0 2023-10-14T11:58:40.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.168+0000 7fd8d4113700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8cc0c6fb0 con 0x7fd8cc0c8bc0 2023-10-14T11:58:40.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.168+0000 7fd8d4113700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8cc0c7f30 con 0x7fd8cc1533d0 2023-10-14T11:58:40.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.168+0000 7fd8d16ae700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd8cc0c8bc0 0x7fd8cc154960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:40.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.168+0000 7fd8d26b0700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd8cc0a70c0 0x7fd8cc154ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50380/0 (socket says 172.21.15.78:50380) 2023-10-14T11:58:40.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.168+0000 7fd8d26b0700 1 -- 172.21.15.78:0/1210138428 learned_addr learned my addr 172.21.15.78:0/1210138428 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:40.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.168+0000 7fd8d1eaf700 1 --2- 172.21.15.78:0/1210138428 >> v2:172.21.15.195:3300/0 conn(0x7fd8cc1533d0 0x7fd8cc0c8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:40.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.168+0000 7fd8d16ae700 1 -- 172.21.15.78:0/1210138428 >> v2:172.21.15.195:3300/0 conn(0x7fd8cc1533d0 msgr2=0x7fd8cc0c8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:40.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.168+0000 7fd8d16ae700 1 --2- 172.21.15.78:0/1210138428 >> v2:172.21.15.195:3300/0 conn(0x7fd8cc1533d0 0x7fd8cc0c8680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:40.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.168+0000 7fd8d16ae700 1 -- 172.21.15.78:0/1210138428 >> v1:172.21.15.78:6789/0 conn(0x7fd8cc0a70c0 legacy=0x7fd8cc154ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:40.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.168+0000 7fd8d16ae700 1 -- 172.21.15.78:0/1210138428 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8cc0c7c50 con 0x7fd8cc0c8bc0 2023-10-14T11:58:40.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.168+0000 7fd8d0ead700 1 -- 172.21.15.78:0/1210138428 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4089883409 0 0) 0x7fd8cc1537b0 con 0x7fd8cc0a70c0 2023-10-14T11:58:40.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.168+0000 7fd8d16ae700 1 --2- 172.21.15.78:0/1210138428 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8cc0c8bc0 0x7fd8cc154960 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc0c6e20 tx=0x7fd8bc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b91ee71253300ded server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:40.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.168+0000 7fd8d0ead700 1 -- 172.21.15.78:0/1210138428 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8bc011830 con 0x7fd8cc0c8bc0 2023-10-14T11:58:40.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.168+0000 7fd8d0ead700 1 -- 172.21.15.78:0/1210138428 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8bc011990 con 0x7fd8cc0c8bc0 2023-10-14T11:58:40.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.169+0000 7fd8d0ead700 1 -- 172.21.15.78:0/1210138428 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8bc011d20 con 0x7fd8cc0c8bc0 2023-10-14T11:58:40.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.169+0000 7fd8d4113700 1 -- 172.21.15.78:0/1210138428 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8cc0c8bc0 msgr2=0x7fd8cc154960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:40.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.169+0000 7fd8d4113700 1 --2- 172.21.15.78:0/1210138428 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8cc0c8bc0 0x7fd8cc154960 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc0c6e20 tx=0x7fd8bc009b50 comp rx=0 tx=0).stop 2023-10-14T11:58:40.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.169+0000 7fd8d4113700 1 -- 172.21.15.78:0/1210138428 shutdown_connections 2023-10-14T11:58:40.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.169+0000 7fd8d4113700 1 --2- 172.21.15.78:0/1210138428 >> v2:172.21.15.195:3300/0 conn(0x7fd8cc1533d0 0x7fd8cc0c8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:40.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.169+0000 7fd8d4113700 1 --2- 172.21.15.78:0/1210138428 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8cc0c8bc0 0x7fd8cc154960 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:40.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.169+0000 7fd8d4113700 1 -- 172.21.15.78:0/1210138428 >> 172.21.15.78:0/1210138428 conn(0x7fd8cc0737d0 msgr2=0x7fd8cc073be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:40.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.169+0000 7fd8d4113700 1 -- 172.21.15.78:0/1210138428 shutdown_connections 2023-10-14T11:58:40.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.169+0000 7fd8d4113700 1 -- 172.21.15.78:0/1210138428 wait complete. 2023-10-14T11:58:40.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.170+0000 7fd8d4113700 1 Processor -- start 2023-10-14T11:58:40.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.170+0000 7fd8d4113700 1 -- start start 2023-10-14T11:58:40.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.171+0000 7fd8d4113700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8cc0a70c0 0x7fd8cc0c3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:40.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.171+0000 7fd8d4113700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd8cc0c8bc0 0x7fd8cc0c5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:40.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.171+0000 7fd8d4113700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8cc1533d0 0x7fd8cc14ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:40.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.171+0000 7fd8d4113700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8cc0c7a70 con 0x7fd8cc1533d0 2023-10-14T11:58:40.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.171+0000 7fd8d4113700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8cc0bb560 con 0x7fd8cc0a70c0 2023-10-14T11:58:40.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.171+0000 7fd8d4113700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8cc1ebb20 con 0x7fd8cc0c8bc0 2023-10-14T11:58:40.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.171+0000 7fd8d26b0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8cc1533d0 0x7fd8cc14ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:40.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.171+0000 7fd8d26b0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8cc1533d0 0x7fd8cc14ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45510/0 (socket says 172.21.15.78:45510) 2023-10-14T11:58:40.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.171+0000 7fd8d26b0700 1 -- 172.21.15.78:0/2220338540 learned_addr learned my addr 172.21.15.78:0/2220338540 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:40.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.171+0000 7fd8d1eaf700 1 --2- 172.21.15.78:0/2220338540 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8cc0a70c0 0x7fd8cc0c3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:40.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.171+0000 7fd8d16ae700 1 --2- 172.21.15.78:0/2220338540 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd8cc0c8bc0 0x7fd8cc0c5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:40.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.171+0000 7fd8d26b0700 1 -- 172.21.15.78:0/2220338540 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd8cc0c8bc0 msgr2=0x7fd8cc0c5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:40.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.171+0000 7fd8d26b0700 1 --2- 172.21.15.78:0/2220338540 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd8cc0c8bc0 0x7fd8cc0c5420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:40.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.171+0000 7fd8d26b0700 1 -- 172.21.15.78:0/2220338540 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8cc0a70c0 msgr2=0x7fd8cc0c3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:40.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.171+0000 7fd8d26b0700 1 --2- 172.21.15.78:0/2220338540 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8cc0a70c0 0x7fd8cc0c3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:40.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.171+0000 7fd8d26b0700 1 -- 172.21.15.78:0/2220338540 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8bc0094a0 con 0x7fd8cc1533d0 2023-10-14T11:58:40.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.171+0000 7fd8d1eaf700 1 --2- 172.21.15.78:0/2220338540 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8cc0a70c0 0x7fd8cc0c3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:40.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.171+0000 7fd8d26b0700 1 --2- 172.21.15.78:0/2220338540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8cc1533d0 0x7fd8cc14ff10 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd8c400cba0 tx=0x7fd8c400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:40.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.172+0000 7fd8c27fc700 1 -- 172.21.15.78:0/2220338540 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c400ab80 con 0x7fd8cc1533d0 2023-10-14T11:58:40.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.172+0000 7fd8c27fc700 1 -- 172.21.15.78:0/2220338540 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8c400ad00 con 0x7fd8cc1533d0 2023-10-14T11:58:40.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.172+0000 7fd8d4113700 1 -- 172.21.15.78:0/2220338540 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8cc0c04d0 con 0x7fd8cc1533d0 2023-10-14T11:58:40.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.172+0000 7fd8c27fc700 1 -- 172.21.15.78:0/2220338540 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c4007310 con 0x7fd8cc1533d0 2023-10-14T11:58:40.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.172+0000 7fd8d4113700 1 -- 172.21.15.78:0/2220338540 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8cc0c4330 con 0x7fd8cc1533d0 2023-10-14T11:58:40.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.173+0000 7fd8d4113700 1 -- 172.21.15.78:0/2220338540 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8cc074b80 con 0x7fd8cc1533d0 2023-10-14T11:58:40.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.173+0000 7fd8c27fc700 1 -- 172.21.15.78:0/2220338540 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd8c4007470 con 0x7fd8cc1533d0 2023-10-14T11:58:40.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.173+0000 7fd8c27fc700 1 --2- 172.21.15.78:0/2220338540 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd8b006af00 0x7fd8b006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:40.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.173+0000 7fd8c27fc700 1 -- 172.21.15.78:0/2220338540 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd8c4089400 con 0x7fd8cc1533d0 2023-10-14T11:58:40.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.173+0000 7fd8d1eaf700 1 --2- 172.21.15.78:0/2220338540 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd8b006af00 0x7fd8b006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:40.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.174+0000 7fd8d1eaf700 1 --2- 172.21.15.78:0/2220338540 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd8b006af00 0x7fd8b006d3c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd8c8009b60 tx=0x7fd8c8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:40.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.178+0000 7fd8c27fc700 1 -- 172.21.15.78:0/2220338540 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd8c4054540 con 0x7fd8cc1533d0 2023-10-14T11:58:40.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.367+0000 7fd8d4113700 1 -- 172.21.15.78:0/2220338540 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd8cc0c70f0 con 0x7fd8cc1533d0 2023-10-14T11:58:40.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.367+0000 7fd8c27fc700 1 -- 172.21.15.78:0/2220338540 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd8c4018020 con 0x7fd8cc1533d0 2023-10-14T11:58:40.370 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2023-10-14T11:58:40.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.370+0000 7fd8d4113700 1 -- 172.21.15.78:0/2220338540 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd8b006af00 msgr2=0x7fd8b006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:40.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.370+0000 7fd8d4113700 1 --2- 172.21.15.78:0/2220338540 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd8b006af00 0x7fd8b006d3c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd8c8009b60 tx=0x7fd8c8008040 comp rx=0 tx=0).stop 2023-10-14T11:58:40.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.370+0000 7fd8d4113700 1 -- 172.21.15.78:0/2220338540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8cc1533d0 msgr2=0x7fd8cc14ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:40.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.370+0000 7fd8d4113700 1 --2- 172.21.15.78:0/2220338540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8cc1533d0 0x7fd8cc14ff10 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd8c400cba0 tx=0x7fd8c400cf60 comp rx=0 tx=0).stop 2023-10-14T11:58:40.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.370+0000 7fd8d4113700 1 -- 172.21.15.78:0/2220338540 shutdown_connections 2023-10-14T11:58:40.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.370+0000 7fd8d4113700 1 --2- 172.21.15.78:0/2220338540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8cc1533d0 0x7fd8cc14ff10 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:40.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.370+0000 7fd8d4113700 1 --2- 172.21.15.78:0/2220338540 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd8b006af00 0x7fd8b006d3c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:40.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.370+0000 7fd8d4113700 1 --2- 172.21.15.78:0/2220338540 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8cc0a70c0 0x7fd8cc0c3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:40.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.370+0000 7fd8d4113700 1 --2- 172.21.15.78:0/2220338540 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd8cc0c8bc0 0x7fd8cc0c5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:40.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.370+0000 7fd8d4113700 1 -- 172.21.15.78:0/2220338540 >> 172.21.15.78:0/2220338540 conn(0x7fd8cc0737d0 msgr2=0x7fd8cc0a9340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:40.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.370+0000 7fd8d4113700 1 -- 172.21.15.78:0/2220338540 shutdown_connections 2023-10-14T11:58:40.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.370+0000 7fd8d4113700 1 -- 172.21.15.78:0/2220338540 wait complete. 2023-10-14T11:58:40.427 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_drop (tasks.cephfs.test_failover.TestClusterAffinity) That unsetting mds_join_fs will not cause failover if no high-affinity standby exists. 2023-10-14T11:58:40.428 INFO:tasks.cephfs_test_runner:test_join_fs_drop (tasks.cephfs.test_failover.TestClusterAffinity) 2023-10-14T11:58:40.429 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T11:58:40.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.548+0000 7f882f4c1700 1 Processor -- start 2023-10-14T11:58:40.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.548+0000 7f882f4c1700 1 -- start start 2023-10-14T11:58:40.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.549+0000 7f882f4c1700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f88281524a0 0x7f882806d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:40.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.549+0000 7f882f4c1700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f882806d7b0 0x7f882814ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:40.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.549+0000 7f882f4c1700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8828151700 con 0x7f88280c1ab0 2023-10-14T11:58:40.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.549+0000 7f882f4c1700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8828155ae0 con 0x7f88281524a0 2023-10-14T11:58:40.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.549+0000 7f882f4c1700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88281519c0 con 0x7f882806d7b0 2023-10-14T11:58:40.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.549+0000 7f882ca5c700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f88281524a0 0x7f882806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:40.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.549+0000 7f882d25d700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f88280c1ab0 0x7f88280a8490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50390/0 (socket says 172.21.15.78:50390) 2023-10-14T11:58:40.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.549+0000 7f882d25d700 1 -- 172.21.15.78:0/2661516795 learned_addr learned my addr 172.21.15.78:0/2661516795 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:40.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.549+0000 7f882da5e700 1 --2- 172.21.15.78:0/2661516795 >> v2:172.21.15.195:3300/0 conn(0x7f882806d7b0 0x7f882814ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:40.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.549+0000 7f882ca5c700 1 -- 172.21.15.78:0/2661516795 >> v2:172.21.15.195:3300/0 conn(0x7f882806d7b0 msgr2=0x7f882814ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:40.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.549+0000 7f882ca5c700 1 --2- 172.21.15.78:0/2661516795 >> v2:172.21.15.195:3300/0 conn(0x7f882806d7b0 0x7f882814ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:40.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.549+0000 7f882ca5c700 1 -- 172.21.15.78:0/2661516795 >> v1:172.21.15.78:6789/0 conn(0x7f88280c1ab0 legacy=0x7f88280a8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:40.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.549+0000 7f881ffff700 1 -- 172.21.15.78:0/2661516795 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2381928307 0 0) 0x7f8828151700 con 0x7f88280c1ab0 2023-10-14T11:58:40.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.549+0000 7f882ca5c700 1 -- 172.21.15.78:0/2661516795 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8828151ca0 con 0x7f88281524a0 2023-10-14T11:58:40.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.550+0000 7f882ca5c700 1 --2- 172.21.15.78:0/2661516795 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88281524a0 0x7f882806d270 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8828155dc0 tx=0x7f882400bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9b4a19e7ced8e95 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:40.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.550+0000 7f881ffff700 1 -- 172.21.15.78:0/2661516795 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882400dc70 con 0x7f88281524a0 2023-10-14T11:58:40.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.550+0000 7f881ffff700 1 -- 172.21.15.78:0/2661516795 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f882400ddd0 con 0x7f88281524a0 2023-10-14T11:58:40.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.550+0000 7f881ffff700 1 -- 172.21.15.78:0/2661516795 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8824010690 con 0x7f88281524a0 2023-10-14T11:58:40.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.550+0000 7f882f4c1700 1 -- 172.21.15.78:0/2661516795 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88281524a0 msgr2=0x7f882806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:40.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.550+0000 7f882f4c1700 1 --2- 172.21.15.78:0/2661516795 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88281524a0 0x7f882806d270 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8828155dc0 tx=0x7f882400bdc0 comp rx=0 tx=0).stop 2023-10-14T11:58:40.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.550+0000 7f882f4c1700 1 -- 172.21.15.78:0/2661516795 shutdown_connections 2023-10-14T11:58:40.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.550+0000 7f882f4c1700 1 --2- 172.21.15.78:0/2661516795 >> v2:172.21.15.195:3300/0 conn(0x7f882806d7b0 0x7f882814ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:40.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.550+0000 7f882f4c1700 1 --2- 172.21.15.78:0/2661516795 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88281524a0 0x7f882806d270 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:40.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.550+0000 7f882f4c1700 1 -- 172.21.15.78:0/2661516795 >> 172.21.15.78:0/2661516795 conn(0x7f88280c2930 msgr2=0x7f88280c2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:40.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.550+0000 7f882f4c1700 1 -- 172.21.15.78:0/2661516795 shutdown_connections 2023-10-14T11:58:40.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.550+0000 7f882f4c1700 1 -- 172.21.15.78:0/2661516795 wait complete. 2023-10-14T11:58:40.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.552+0000 7f882f4c1700 1 Processor -- start 2023-10-14T11:58:40.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.552+0000 7f882f4c1700 1 -- start start 2023-10-14T11:58:40.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.552+0000 7f882f4c1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f882806d7b0 0x7f8828153f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:40.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.552+0000 7f882f4c1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88280c1ab0 0x7f8828147310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:40.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.552+0000 7f882f4c1700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f88281524a0 0x7f88281528c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:40.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.552+0000 7f882f4c1700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88281546c0 con 0x7f88280c1ab0 2023-10-14T11:58:40.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.552+0000 7f882f4c1700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88280bfd70 con 0x7f882806d7b0 2023-10-14T11:58:40.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.552+0000 7f882f4c1700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8828153ae0 con 0x7f88281524a0 2023-10-14T11:58:40.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.552+0000 7f882ca5c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88280c1ab0 0x7f8828147310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:40.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.552+0000 7f882ca5c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88280c1ab0 0x7f8828147310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45522/0 (socket says 172.21.15.78:45522) 2023-10-14T11:58:40.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.552+0000 7f882ca5c700 1 -- 172.21.15.78:0/2317966734 learned_addr learned my addr 172.21.15.78:0/2317966734 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:40.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.552+0000 7f882da5e700 1 --2- 172.21.15.78:0/2317966734 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f88281524a0 0x7f88281528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:40.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.552+0000 7f882d25d700 1 --2- 172.21.15.78:0/2317966734 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f882806d7b0 0x7f8828153f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:40.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.553+0000 7f882ca5c700 1 -- 172.21.15.78:0/2317966734 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f88281524a0 msgr2=0x7f88281528c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:40.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.553+0000 7f882ca5c700 1 --2- 172.21.15.78:0/2317966734 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f88281524a0 0x7f88281528c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:40.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.553+0000 7f882ca5c700 1 -- 172.21.15.78:0/2317966734 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f882806d7b0 msgr2=0x7f8828153f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:40.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.553+0000 7f882ca5c700 1 --2- 172.21.15.78:0/2317966734 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f882806d7b0 0x7f8828153f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:40.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.553+0000 7f882ca5c700 1 -- 172.21.15.78:0/2317966734 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f882400b740 con 0x7f88280c1ab0 2023-10-14T11:58:40.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.553+0000 7f882da5e700 1 --2- 172.21.15.78:0/2317966734 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f88281524a0 0x7f88281528c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:40.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.553+0000 7f882d25d700 1 --2- 172.21.15.78:0/2317966734 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f882806d7b0 0x7f8828153f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:40.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.553+0000 7f882ca5c700 1 --2- 172.21.15.78:0/2317966734 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88280c1ab0 0x7f8828147310 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f8824015000 tx=0x7f8824004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:40.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.553+0000 7f881dffb700 1 -- 172.21.15.78:0/2317966734 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882401f070 con 0x7f88280c1ab0 2023-10-14T11:58:40.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.553+0000 7f881dffb700 1 -- 172.21.15.78:0/2317966734 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8824003a70 con 0x7f88280c1ab0 2023-10-14T11:58:40.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.553+0000 7f882f4c1700 1 -- 172.21.15.78:0/2317966734 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88281e9c80 con 0x7f88280c1ab0 2023-10-14T11:58:40.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.553+0000 7f881dffb700 1 -- 172.21.15.78:0/2317966734 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8824003e20 con 0x7f88280c1ab0 2023-10-14T11:58:40.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.553+0000 7f882f4c1700 1 -- 172.21.15.78:0/2317966734 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88280c02a0 con 0x7f88280c1ab0 2023-10-14T11:58:40.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.554+0000 7f882f4c1700 1 -- 172.21.15.78:0/2317966734 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8828074b80 con 0x7f88280c1ab0 2023-10-14T11:58:40.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.554+0000 7f881dffb700 1 -- 172.21.15.78:0/2317966734 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8824019400 con 0x7f88280c1ab0 2023-10-14T11:58:40.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.555+0000 7f881dffb700 1 --2- 172.21.15.78:0/2317966734 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f880c06af00 0x7f880c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:40.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.555+0000 7f881dffb700 1 -- 172.21.15.78:0/2317966734 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f882408c9a0 con 0x7f88280c1ab0 2023-10-14T11:58:40.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.555+0000 7f882d25d700 1 --2- 172.21.15.78:0/2317966734 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f880c06af00 0x7f880c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:40.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.555+0000 7f882d25d700 1 --2- 172.21.15.78:0/2317966734 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f880c06af00 0x7f880c06d3c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8818000f60 tx=0x7f8818007ca0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:40.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.559+0000 7f881dffb700 1 -- 172.21.15.78:0/2317966734 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8824057a60 con 0x7f88280c1ab0 2023-10-14T11:58:40.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.748+0000 7f882f4c1700 1 -- 172.21.15.78:0/2317966734 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8828153c20 con 0x7f88280c1ab0 2023-10-14T11:58:40.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.749+0000 7f881dffb700 1 -- 172.21.15.78:0/2317966734 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16481 (secure 0 0 0) 0x7f882405b080 con 0x7f88280c1ab0 2023-10-14T11:58:40.751 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:40.752 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":21,"fsid":"4c5dc134-3092-406a-a1e1-13e6d9e926e8","created":"2023-10-14T11:57:57.363750+0000","modified":"2023-10-14T11:58:16.139413+0000","last_up_change":"2023-10-14T11:58:06.982472+0000","last_in_change":"2023-10-14T11:57:59.821373+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T11:58:09.001200+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T11:58:12.778573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T11:58:13.490286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"625a5fc0-9474-4eae-b80e-14c27a74ad19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6801","nonce":3884969780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6803","nonce":3884969780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6807","nonce":3884969780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6805","nonce":3884969780}]},"public_addr":"172.21.15.78:6801/3884969780","cluster_addr":"172.21.15.78:6803/3884969780","heartbeat_back_addr":"172.21.15.78:6807/3884969780","heartbeat_front_addr":"172.21.15.78:6805/3884969780","state":["exists","up"]},{"osd":1,"uuid":"49d63db1-2981-49ab-9c40-c50fc744b982","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6809","nonce":2176928406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6811","nonce":2176928406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6815","nonce":2176928406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6813","nonce":2176928406}]},"public_addr":"172.21.15.78:6809/2176928406","cluster_addr":"172.21.15.78:6811/2176928406","heartbeat_back_addr":"172.21.15.78:6815/2176928406","heartbeat_front_addr":"172.21.15.78:6813/2176928406","state":["exists","up"]},{"osd":2,"uuid":"1d7d0b29-cbd1-4717-9fe6-765a8d4998b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6825","nonce":3647044577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6827","nonce":3647044577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6831","nonce":3647044577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6829","nonce":3647044577}]},"public_addr":"172.21.15.78:6825/3647044577","cluster_addr":"172.21.15.78:6827/3647044577","heartbeat_back_addr":"172.21.15.78:6831/3647044577","heartbeat_front_addr":"172.21.15.78:6829/3647044577","state":["exists","up"]},{"osd":3,"uuid":"2f75ded1-e9e2-40e7-99ae-45e1e425e1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6817","nonce":3584112288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6819","nonce":3584112288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6823","nonce":3584112288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6821","nonce":3584112288}]},"public_addr":"172.21.15.78:6817/3584112288","cluster_addr":"172.21.15.78:6819/3584112288","heartbeat_back_addr":"172.21.15.78:6823/3584112288","heartbeat_front_addr":"172.21.15.78:6821/3584112288","state":["exists","up"]},{"osd":4,"uuid":"5130b432-8358-46cb-8a31-3064c976f6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6801","nonce":3732756301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6803","nonce":3732756301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6807","nonce":3732756301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6805","nonce":3732756301}]},"public_addr":"172.21.15.195:6801/3732756301","cluster_addr":"172.21.15.195:6803/3732756301","heartbeat_back_addr":"172.21.15.195:6807/3732756301","heartbeat_front_addr":"172.21.15.195:6805/3732756301","state":["exists","up"]},{"osd":5,"uuid":"60e234a9-a506-4e40-9c19-49a58f3cf28c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6825","nonce":2857706411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6827","nonce":2857706411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6831","nonce":2857706411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6829","nonce":2857706411}]},"public_addr":"172.21.15.195:6825/2857706411","cluster_addr":"172.21.15.195:6827/2857706411","heartbeat_back_addr":"172.21.15.195:6831/2857706411","heartbeat_front_addr":"172.21.15.195:6829/2857706411","state":["exists","up"]},{"osd":6,"uuid":"df7d2dd6-26b6-4eb1-96e5-94d1f3955b33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6810","nonce":72554374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6813","nonce":72554374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6820","nonce":72554374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6817","nonce":72554374}]},"public_addr":"172.21.15.195:6810/72554374","cluster_addr":"172.21.15.195:6813/72554374","heartbeat_back_addr":"172.21.15.195:6820/72554374","heartbeat_front_addr":"172.21.15.195:6817/72554374","state":["exists","up"]},{"osd":7,"uuid":"47a9d4de-f56d-4aa4-9900-c4a9cf9b2f97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6809","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6811","nonce":482472979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6816","nonce":482472979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6823","nonce":482472979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6821","nonce":482472979}]},"public_addr":"172.21.15.195:6811/482472979","cluster_addr":"172.21.15.195:6816/482472979","heartbeat_back_addr":"172.21.15.195:6823/482472979","heartbeat_front_addr":"172.21.15.195:6821/482472979","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.754003+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.005994+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.043603+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.024506+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.062585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.033767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.994825+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.038300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T11:58:40.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.751+0000 7f882f4c1700 1 -- 172.21.15.78:0/2317966734 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f880c06af00 msgr2=0x7f880c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:40.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.751+0000 7f882f4c1700 1 --2- 172.21.15.78:0/2317966734 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f880c06af00 0x7f880c06d3c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8818000f60 tx=0x7f8818007ca0 comp rx=0 tx=0).stop 2023-10-14T11:58:40.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.751+0000 7f882f4c1700 1 -- 172.21.15.78:0/2317966734 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88280c1ab0 msgr2=0x7f8828147310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:40.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.751+0000 7f882f4c1700 1 --2- 172.21.15.78:0/2317966734 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88280c1ab0 0x7f8828147310 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f8824015000 tx=0x7f8824004750 comp rx=0 tx=0).stop 2023-10-14T11:58:40.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.751+0000 7f882f4c1700 1 -- 172.21.15.78:0/2317966734 shutdown_connections 2023-10-14T11:58:40.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.751+0000 7f882f4c1700 1 --2- 172.21.15.78:0/2317966734 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f88281524a0 0x7f88281528c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:40.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.751+0000 7f882f4c1700 1 --2- 172.21.15.78:0/2317966734 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f880c06af00 0x7f880c06d3c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:40.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.751+0000 7f882f4c1700 1 --2- 172.21.15.78:0/2317966734 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f882806d7b0 0x7f8828153f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:40.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.751+0000 7f882f4c1700 1 --2- 172.21.15.78:0/2317966734 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88280c1ab0 0x7f8828147310 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:40.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.751+0000 7f882f4c1700 1 -- 172.21.15.78:0/2317966734 >> 172.21.15.78:0/2317966734 conn(0x7f88280c2930 msgr2=0x7f882814aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:40.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.752+0000 7f882f4c1700 1 -- 172.21.15.78:0/2317966734 shutdown_connections 2023-10-14T11:58:40.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.752+0000 7f882f4c1700 1 -- 172.21.15.78:0/2317966734 wait complete. 2023-10-14T11:58:40.766 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T11:58:09.001200+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T11:58:12.778573+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T11:58:13.490286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T11:58:40.767 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T11:58:40.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.886+0000 7fd069bc1700 1 Processor -- start 2023-10-14T11:58:40.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.887+0000 7fd069bc1700 1 -- start start 2023-10-14T11:58:40.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.887+0000 7fd069bc1700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd06413dbe0 0x7fd06406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:40.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.887+0000 7fd069bc1700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd0640c4540 0x7fd0640c02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:40.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.887+0000 7fd069bc1700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd06406ebe0 con 0x7fd06406ae10 2023-10-14T11:58:40.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.887+0000 7fd069bc1700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd06406d330 con 0x7fd06413dbe0 2023-10-14T11:58:40.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.887+0000 7fd069bc1700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0641558c0 con 0x7fd0640c4540 2023-10-14T11:58:40.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.887+0000 7fd0637fe700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd06413dbe0 0x7fd06406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:40.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.887+0000 7fd0637fe700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd06413dbe0 0x7fd06406a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45528/0 (socket says 172.21.15.78:45528) 2023-10-14T11:58:40.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.887+0000 7fd0637fe700 1 -- 172.21.15.78:0/1983864330 learned_addr learned my addr 172.21.15.78:0/1983864330 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:40.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.888+0000 7fd062ffd700 1 --2- 172.21.15.78:0/1983864330 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd0640c4540 0x7fd0640c02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:40.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.888+0000 7fd0637fe700 1 -- 172.21.15.78:0/1983864330 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd0640c4540 msgr2=0x7fd0640c02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:40.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.888+0000 7fd0637fe700 1 --2- 172.21.15.78:0/1983864330 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd0640c4540 0x7fd0640c02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:40.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.888+0000 7fd0637fe700 1 -- 172.21.15.78:0/1983864330 >> v1:172.21.15.195:6789/0 conn(0x7fd06406ae10 legacy=0x7fd0640c0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:40.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.888+0000 7fd0637fe700 1 -- 172.21.15.78:0/1983864330 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd064154b00 con 0x7fd06413dbe0 2023-10-14T11:58:40.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.888+0000 7fd0637fe700 1 --2- 172.21.15.78:0/1983864330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd06413dbe0 0x7fd06406a8d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd0540098f0 tx=0x7fd054009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffeed1e7b0763db9 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:40.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.888+0000 7fd0627fc700 1 -- 172.21.15.78:0/1983864330 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05400b8a0 con 0x7fd06413dbe0 2023-10-14T11:58:40.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.888+0000 7fd0627fc700 1 -- 172.21.15.78:0/1983864330 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd05400ba00 con 0x7fd06413dbe0 2023-10-14T11:58:40.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.888+0000 7fd0627fc700 1 -- 172.21.15.78:0/1983864330 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05400bd90 con 0x7fd06413dbe0 2023-10-14T11:58:40.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.889+0000 7fd069bc1700 1 -- 172.21.15.78:0/1983864330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd06413dbe0 msgr2=0x7fd06406a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:40.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.889+0000 7fd069bc1700 1 --2- 172.21.15.78:0/1983864330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd06413dbe0 0x7fd06406a8d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd0540098f0 tx=0x7fd054009c00 comp rx=0 tx=0).stop 2023-10-14T11:58:40.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.889+0000 7fd069bc1700 1 -- 172.21.15.78:0/1983864330 shutdown_connections 2023-10-14T11:58:40.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.889+0000 7fd069bc1700 1 --2- 172.21.15.78:0/1983864330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd06413dbe0 0x7fd06406a8d0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:40.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.889+0000 7fd069bc1700 1 --2- 172.21.15.78:0/1983864330 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd0640c4540 0x7fd0640c02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:40.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.889+0000 7fd069bc1700 1 -- 172.21.15.78:0/1983864330 >> 172.21.15.78:0/1983864330 conn(0x7fd0640c3a50 msgr2=0x7fd0640c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:40.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.889+0000 7fd069bc1700 1 -- 172.21.15.78:0/1983864330 shutdown_connections 2023-10-14T11:58:40.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.889+0000 7fd069bc1700 1 -- 172.21.15.78:0/1983864330 wait complete. 2023-10-14T11:58:40.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.890+0000 7fd069bc1700 1 Processor -- start 2023-10-14T11:58:40.899 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.890+0000 7fd069bc1700 1 -- start start 2023-10-14T11:58:40.899 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.891+0000 7fd069bc1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd06406ae10 0x7fd064158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:40.899 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.891+0000 7fd069bc1700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd0640c4540 0x7fd064139960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:40.899 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.891+0000 7fd069bc1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd06413dbe0 0x7fd064155df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:40.899 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.891+0000 7fd069bc1700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0640a95e0 con 0x7fd06413dbe0 2023-10-14T11:58:40.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.891+0000 7fd069bc1700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0641346b0 con 0x7fd06406ae10 2023-10-14T11:58:40.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.891+0000 7fd069bc1700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0641edf70 con 0x7fd0640c4540 2023-10-14T11:58:40.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.891+0000 7fd063fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd06413dbe0 0x7fd064155df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:40.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.891+0000 7fd063fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd06413dbe0 0x7fd064155df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45538/0 (socket says 172.21.15.78:45538) 2023-10-14T11:58:40.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.891+0000 7fd063fff700 1 -- 172.21.15.78:0/3661095632 learned_addr learned my addr 172.21.15.78:0/3661095632 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.891+0000 7fd0637fe700 1 --2- 172.21.15.78:0/3661095632 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd06406ae10 0x7fd064158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.891+0000 7fd062ffd700 1 --2- 172.21.15.78:0/3661095632 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd0640c4540 0x7fd064139960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.891+0000 7fd063fff700 1 -- 172.21.15.78:0/3661095632 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd0640c4540 msgr2=0x7fd064139960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.891+0000 7fd063fff700 1 --2- 172.21.15.78:0/3661095632 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd0640c4540 0x7fd064139960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.891+0000 7fd063fff700 1 -- 172.21.15.78:0/3661095632 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd06406ae10 msgr2=0x7fd064158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:40.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.891+0000 7fd063fff700 1 --2- 172.21.15.78:0/3661095632 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd06406ae10 0x7fd064158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:40.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.891+0000 7fd063fff700 1 -- 172.21.15.78:0/3661095632 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd054009550 con 0x7fd06413dbe0 2023-10-14T11:58:40.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.891+0000 7fd063fff700 1 --2- 172.21.15.78:0/3661095632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd06413dbe0 0x7fd064155df0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd05c00cb40 tx=0x7fd05c00ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:40.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.891+0000 7fd05bfff700 1 -- 172.21.15.78:0/3661095632 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05c003c00 con 0x7fd06413dbe0 2023-10-14T11:58:40.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.891+0000 7fd05bfff700 1 -- 172.21.15.78:0/3661095632 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd05c003d60 con 0x7fd06413dbe0 2023-10-14T11:58:40.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.891+0000 7fd069bc1700 1 -- 172.21.15.78:0/3661095632 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0641ed9f0 con 0x7fd06413dbe0 2023-10-14T11:58:40.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.891+0000 7fd05bfff700 1 -- 172.21.15.78:0/3661095632 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05c00aa20 con 0x7fd06413dbe0 2023-10-14T11:58:40.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.891+0000 7fd069bc1700 1 -- 172.21.15.78:0/3661095632 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd064152a80 con 0x7fd06413dbe0 2023-10-14T11:58:40.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.892+0000 7fd069bc1700 1 -- 172.21.15.78:0/3661095632 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd064155b10 con 0x7fd06413dbe0 2023-10-14T11:58:40.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.893+0000 7fd05bfff700 1 -- 172.21.15.78:0/3661095632 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd05c00ab80 con 0x7fd06413dbe0 2023-10-14T11:58:40.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.893+0000 7fd05bfff700 1 --2- 172.21.15.78:0/3661095632 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd04406af00 0x7fd04406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:40.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.893+0000 7fd05bfff700 1 -- 172.21.15.78:0/3661095632 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd05c089480 con 0x7fd06413dbe0 2023-10-14T11:58:40.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.893+0000 7fd0637fe700 1 --2- 172.21.15.78:0/3661095632 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd04406af00 0x7fd04406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:40.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.894+0000 7fd0637fe700 1 --2- 172.21.15.78:0/3661095632 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd04406af00 0x7fd04406d3c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd054014040 tx=0x7fd054008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:40.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:40.898+0000 7fd05bfff700 1 -- 172.21.15.78:0/3661095632 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd064155b10 con 0x7fd06413dbe0 2023-10-14T11:58:41.086 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.085+0000 7fd069bc1700 1 -- 172.21.15.78:0/3661095632 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd064155b10 con 0x7fd06413dbe0 2023-10-14T11:58:41.087 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.085+0000 7fd05bfff700 1 -- 172.21.15.78:0/3661095632 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd064155b10 con 0x7fd06413dbe0 2023-10-14T11:58:41.087 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-10-14T11:58:41.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.087+0000 7fd069bc1700 1 -- 172.21.15.78:0/3661095632 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd04406af00 msgr2=0x7fd04406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:41.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.087+0000 7fd069bc1700 1 --2- 172.21.15.78:0/3661095632 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd04406af00 0x7fd04406d3c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd054014040 tx=0x7fd054008030 comp rx=0 tx=0).stop 2023-10-14T11:58:41.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.087+0000 7fd069bc1700 1 -- 172.21.15.78:0/3661095632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd06413dbe0 msgr2=0x7fd064155df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:41.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.087+0000 7fd069bc1700 1 --2- 172.21.15.78:0/3661095632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd06413dbe0 0x7fd064155df0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd05c00cb40 tx=0x7fd05c00ce50 comp rx=0 tx=0).stop 2023-10-14T11:58:41.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.087+0000 7fd069bc1700 1 -- 172.21.15.78:0/3661095632 shutdown_connections 2023-10-14T11:58:41.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.087+0000 7fd069bc1700 1 --2- 172.21.15.78:0/3661095632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd06413dbe0 0x7fd064155df0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:41.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.087+0000 7fd069bc1700 1 --2- 172.21.15.78:0/3661095632 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd04406af00 0x7fd04406d3c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:41.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.087+0000 7fd069bc1700 1 --2- 172.21.15.78:0/3661095632 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd06406ae10 0x7fd064158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:41.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.087+0000 7fd069bc1700 1 --2- 172.21.15.78:0/3661095632 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd0640c4540 0x7fd064139960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:41.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.087+0000 7fd069bc1700 1 -- 172.21.15.78:0/3661095632 >> 172.21.15.78:0/3661095632 conn(0x7fd0640c3a50 msgr2=0x7fd06406ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:41.092 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.087+0000 7fd069bc1700 1 -- 172.21.15.78:0/3661095632 shutdown_connections 2023-10-14T11:58:41.092 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.087+0000 7fd069bc1700 1 -- 172.21.15.78:0/3661095632 wait complete. 2023-10-14T11:58:41.101 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T11:58:41.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.222+0000 7fd6245bd700 1 Processor -- start 2023-10-14T11:58:41.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.222+0000 7fd6245bd700 1 -- start start 2023-10-14T11:58:41.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.223+0000 7fd6245bd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd61c0c35e0 0x7fd61c1535d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:41.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.223+0000 7fd6245bd700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd61c154c20 0x7fd61c155070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:41.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.223+0000 7fd6245bd700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd61c0c26d0 con 0x7fd61c14ed10 2023-10-14T11:58:41.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.223+0000 7fd6245bd700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd61c0a8520 con 0x7fd61c0c35e0 2023-10-14T11:58:41.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.223+0000 7fd6245bd700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd61c0c29b0 con 0x7fd61c154c20 2023-10-14T11:58:41.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.223+0000 7fd622b5a700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd61c154c20 0x7fd61c155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:41.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.223+0000 7fd621b58700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd61c0c35e0 0x7fd61c1535d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:41.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.223+0000 7fd622b5a700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd61c154c20 0x7fd61c155070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.78:45216/0 (socket says 172.21.15.78:45216) 2023-10-14T11:58:41.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.223+0000 7fd622b5a700 1 -- 172.21.15.78:0/413079407 learned_addr learned my addr 172.21.15.78:0/413079407 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:41.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.224+0000 7fd622b5a700 1 -- 172.21.15.78:0/413079407 >> v2:172.21.15.195:3300/0 conn(0x7fd61c0c35e0 msgr2=0x7fd61c1535d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:41.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.224+0000 7fd622b5a700 1 --2- 172.21.15.78:0/413079407 >> v2:172.21.15.195:3300/0 conn(0x7fd61c0c35e0 0x7fd61c1535d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:41.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.224+0000 7fd622b5a700 1 -- 172.21.15.78:0/413079407 >> v1:172.21.15.195:6789/0 conn(0x7fd61c14ed10 legacy=0x7fd61c152ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:41.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.224+0000 7fd622b5a700 1 -- 172.21.15.78:0/413079407 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd61c0c2c90 con 0x7fd61c154c20 2023-10-14T11:58:41.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.224+0000 7fd621b58700 1 --2- 172.21.15.78:0/413079407 >> v2:172.21.15.195:3300/0 conn(0x7fd61c0c35e0 0x7fd61c1535d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T11:58:41.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.224+0000 7fd622b5a700 1 --2- 172.21.15.78:0/413079407 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd61c154c20 0x7fd61c155070 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd614009e50 tx=0x7fd61400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f435a3020a05c3f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:41.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.224+0000 7fd621357700 1 -- 172.21.15.78:0/413079407 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd61400e070 con 0x7fd61c154c20 2023-10-14T11:58:41.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.224+0000 7fd621357700 1 -- 172.21.15.78:0/413079407 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd61400b910 con 0x7fd61c154c20 2023-10-14T11:58:41.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.224+0000 7fd621357700 1 -- 172.21.15.78:0/413079407 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd61400bcc0 con 0x7fd61c154c20 2023-10-14T11:58:41.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.225+0000 7fd6245bd700 1 -- 172.21.15.78:0/413079407 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd61c154c20 msgr2=0x7fd61c155070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:41.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.225+0000 7fd6245bd700 1 --2- 172.21.15.78:0/413079407 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd61c154c20 0x7fd61c155070 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd614009e50 tx=0x7fd61400b330 comp rx=0 tx=0).stop 2023-10-14T11:58:41.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.225+0000 7fd6245bd700 1 -- 172.21.15.78:0/413079407 shutdown_connections 2023-10-14T11:58:41.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.225+0000 7fd6245bd700 1 --2- 172.21.15.78:0/413079407 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd61c154c20 0x7fd61c155070 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:41.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.225+0000 7fd6245bd700 1 --2- 172.21.15.78:0/413079407 >> v2:172.21.15.195:3300/0 conn(0x7fd61c0c35e0 0x7fd61c1535d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:41.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.225+0000 7fd6245bd700 1 -- 172.21.15.78:0/413079407 >> 172.21.15.78:0/413079407 conn(0x7fd61c0c3a50 msgr2=0x7fd61c0c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:41.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.225+0000 7fd6245bd700 1 -- 172.21.15.78:0/413079407 shutdown_connections 2023-10-14T11:58:41.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.225+0000 7fd6245bd700 1 -- 172.21.15.78:0/413079407 wait complete. 2023-10-14T11:58:41.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.226+0000 7fd6245bd700 1 Processor -- start 2023-10-14T11:58:41.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.226+0000 7fd6245bd700 1 -- start start 2023-10-14T11:58:41.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.227+0000 7fd6245bd700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd61c14ed10 0x7fd61c0c0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:41.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.227+0000 7fd6245bd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd61c154c20 0x7fd61c1ed4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:41.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.227+0000 7fd6245bd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd61c0c6ad0 0x7fd61c1ed9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:41.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.227+0000 7fd6245bd700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd61c0ab2b0 con 0x7fd61c0c6ad0 2023-10-14T11:58:41.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.227+0000 7fd6245bd700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd61c1ea3b0 con 0x7fd61c154c20 2023-10-14T11:58:41.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.227+0000 7fd6245bd700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd61c1ed060 con 0x7fd61c14ed10 2023-10-14T11:58:41.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.227+0000 7fd622b5a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd61c0c6ad0 0x7fd61c1ed9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:41.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.227+0000 7fd622b5a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd61c0c6ad0 0x7fd61c1ed9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45552/0 (socket says 172.21.15.78:45552) 2023-10-14T11:58:41.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.227+0000 7fd622b5a700 1 -- 172.21.15.78:0/4047440464 learned_addr learned my addr 172.21.15.78:0/4047440464 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:41.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.227+0000 7fd621b58700 1 --2- 172.21.15.78:0/4047440464 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd61c154c20 0x7fd61c1ed4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:41.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.227+0000 7fd622359700 1 --2- 172.21.15.78:0/4047440464 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd61c14ed10 0x7fd61c0c0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:41.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.227+0000 7fd622b5a700 1 -- 172.21.15.78:0/4047440464 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd61c14ed10 msgr2=0x7fd61c0c0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:41.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.227+0000 7fd622b5a700 1 --2- 172.21.15.78:0/4047440464 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd61c14ed10 0x7fd61c0c0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:41.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.227+0000 7fd622b5a700 1 -- 172.21.15.78:0/4047440464 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd61c154c20 msgr2=0x7fd61c1ed4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:41.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.227+0000 7fd622b5a700 1 --2- 172.21.15.78:0/4047440464 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd61c154c20 0x7fd61c1ed4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:41.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.227+0000 7fd622b5a700 1 -- 172.21.15.78:0/4047440464 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6180094a0 con 0x7fd61c0c6ad0 2023-10-14T11:58:41.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.227+0000 7fd622b5a700 1 --2- 172.21.15.78:0/4047440464 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd61c0c6ad0 0x7fd61c1ed9f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd61c0c2af0 tx=0x7fd614004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:41.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.227+0000 7fd612ffd700 1 -- 172.21.15.78:0/4047440464 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd614004800 con 0x7fd61c0c6ad0 2023-10-14T11:58:41.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.227+0000 7fd612ffd700 1 -- 172.21.15.78:0/4047440464 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd614004960 con 0x7fd61c0c6ad0 2023-10-14T11:58:41.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.227+0000 7fd6245bd700 1 -- 172.21.15.78:0/4047440464 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd614009b30 con 0x7fd61c0c6ad0 2023-10-14T11:58:41.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.227+0000 7fd612ffd700 1 -- 172.21.15.78:0/4047440464 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd61401b670 con 0x7fd61c0c6ad0 2023-10-14T11:58:41.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.227+0000 7fd6245bd700 1 -- 172.21.15.78:0/4047440464 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd61c1ea5a0 con 0x7fd61c0c6ad0 2023-10-14T11:58:41.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.228+0000 7fd6245bd700 1 -- 172.21.15.78:0/4047440464 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd61c074b80 con 0x7fd61c0c6ad0 2023-10-14T11:58:41.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.229+0000 7fd612ffd700 1 -- 172.21.15.78:0/4047440464 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd61401b7d0 con 0x7fd61c0c6ad0 2023-10-14T11:58:41.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.229+0000 7fd612ffd700 1 --2- 172.21.15.78:0/4047440464 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd60006af00 0x7fd60006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:41.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.229+0000 7fd612ffd700 1 -- 172.21.15.78:0/4047440464 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd61408ac80 con 0x7fd61c0c6ad0 2023-10-14T11:58:41.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.229+0000 7fd622359700 1 --2- 172.21.15.78:0/4047440464 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd60006af00 0x7fd60006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:41.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.230+0000 7fd622359700 1 --2- 172.21.15.78:0/4047440464 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd60006af00 0x7fd60006d3c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd60c007fd0 tx=0x7fd60c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:41.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.234+0000 7fd612ffd700 1 -- 172.21.15.78:0/4047440464 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd614055dc0 con 0x7fd61c0c6ad0 2023-10-14T11:58:41.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.424+0000 7fd6245bd700 1 -- 172.21.15.78:0/4047440464 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd61c0c2330 con 0x7fd61c0c6ad0 2023-10-14T11:58:41.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.424+0000 7fd612ffd700 1 -- 172.21.15.78:0/4047440464 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd6140593e0 con 0x7fd61c0c6ad0 2023-10-14T11:58:41.426 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-10-14T11:58:41.428 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.426+0000 7fd6245bd700 1 -- 172.21.15.78:0/4047440464 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd60006af00 msgr2=0x7fd60006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:41.428 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.426+0000 7fd6245bd700 1 --2- 172.21.15.78:0/4047440464 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd60006af00 0x7fd60006d3c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd60c007fd0 tx=0x7fd60c009040 comp rx=0 tx=0).stop 2023-10-14T11:58:41.429 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.426+0000 7fd6245bd700 1 -- 172.21.15.78:0/4047440464 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd61c0c6ad0 msgr2=0x7fd61c1ed9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:41.429 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.426+0000 7fd6245bd700 1 --2- 172.21.15.78:0/4047440464 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd61c0c6ad0 0x7fd61c1ed9f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd61c0c2af0 tx=0x7fd614004ad0 comp rx=0 tx=0).stop 2023-10-14T11:58:41.429 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.426+0000 7fd6245bd700 1 -- 172.21.15.78:0/4047440464 shutdown_connections 2023-10-14T11:58:41.429 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.426+0000 7fd6245bd700 1 --2- 172.21.15.78:0/4047440464 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd61c0c6ad0 0x7fd61c1ed9f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:41.429 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.426+0000 7fd6245bd700 1 --2- 172.21.15.78:0/4047440464 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd61c14ed10 0x7fd61c0c0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:41.430 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.426+0000 7fd6245bd700 1 --2- 172.21.15.78:0/4047440464 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd60006af00 0x7fd60006d3c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:41.430 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.426+0000 7fd6245bd700 1 --2- 172.21.15.78:0/4047440464 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd61c154c20 0x7fd61c1ed4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:41.430 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.426+0000 7fd6245bd700 1 -- 172.21.15.78:0/4047440464 >> 172.21.15.78:0/4047440464 conn(0x7fd61c0c3a50 msgr2=0x7fd61c06d780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:41.430 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.426+0000 7fd6245bd700 1 -- 172.21.15.78:0/4047440464 shutdown_connections 2023-10-14T11:58:41.431 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.426+0000 7fd6245bd700 1 -- 172.21.15.78:0/4047440464 wait complete. 2023-10-14T11:58:41.440 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T11:58:41.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.561+0000 7ffb53410700 1 Processor -- start 2023-10-14T11:58:41.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.561+0000 7ffb53410700 1 -- start start 2023-10-14T11:58:41.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.562+0000 7ffb53410700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffb4c13dbe0 0x7ffb4c06a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:41.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.562+0000 7ffb53410700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ffb4c0c4540 0x7ffb4c0c02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:41.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.562+0000 7ffb53410700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ffb4c06ae10 0x7ffb4c0c0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:41.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.562+0000 7ffb53410700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb4c06d330 con 0x7ffb4c0c4540 2023-10-14T11:58:41.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.562+0000 7ffb53410700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb4c06ebe0 con 0x7ffb4c06ae10 2023-10-14T11:58:41.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.562+0000 7ffb53410700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c0c0d60 con 0x7ffb4c13dbe0 2023-10-14T11:58:41.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.562+0000 7ffb509ab700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ffb4c0c4540 0x7ffb4c0c02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:41.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.562+0000 7ffb509ab700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ffb4c0c4540 0x7ffb4c0c02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45556/0 (socket says 172.21.15.78:45556) 2023-10-14T11:58:41.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.562+0000 7ffb509ab700 1 -- 172.21.15.78:0/3876327572 learned_addr learned my addr 172.21.15.78:0/3876327572 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:41.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.562+0000 7ffb511ac700 1 --2- 172.21.15.78:0/3876327572 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffb4c13dbe0 0x7ffb4c06a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:41.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.562+0000 7ffb519ad700 1 --2- 172.21.15.78:0/3876327572 >> v2:172.21.15.195:3300/0 conn(0x7ffb4c06ae10 0x7ffb4c0c0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:41.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.562+0000 7ffb509ab700 1 -- 172.21.15.78:0/3876327572 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffb4c13dbe0 msgr2=0x7ffb4c06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:41.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.562+0000 7ffb509ab700 1 --2- 172.21.15.78:0/3876327572 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffb4c13dbe0 0x7ffb4c06a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:41.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.562+0000 7ffb509ab700 1 -- 172.21.15.78:0/3876327572 >> v2:172.21.15.195:3300/0 conn(0x7ffb4c06ae10 msgr2=0x7ffb4c0c0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:41.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.562+0000 7ffb509ab700 1 --2- 172.21.15.78:0/3876327572 >> v2:172.21.15.195:3300/0 conn(0x7ffb4c06ae10 0x7ffb4c0c0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:41.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.562+0000 7ffb509ab700 1 -- 172.21.15.78:0/3876327572 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb4c1558c0 con 0x7ffb4c0c4540 2023-10-14T11:58:41.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.562+0000 7ffb519ad700 1 --2- 172.21.15.78:0/3876327572 >> v2:172.21.15.195:3300/0 conn(0x7ffb4c06ae10 0x7ffb4c0c0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:41.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.562+0000 7ffb509ab700 1 --2- 172.21.15.78:0/3876327572 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb4c0c4540 0x7ffb4c0c02e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c06e270 tx=0x7ffb48009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=256c3917b28030fa server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:41.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.563+0000 7ffb43fff700 1 -- 172.21.15.78:0/3876327572 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4800b870 con 0x7ffb4c0c4540 2023-10-14T11:58:41.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.563+0000 7ffb43fff700 1 -- 172.21.15.78:0/3876327572 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb4800b9d0 con 0x7ffb4c0c4540 2023-10-14T11:58:41.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.563+0000 7ffb43fff700 1 -- 172.21.15.78:0/3876327572 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4800bd60 con 0x7ffb4c0c4540 2023-10-14T11:58:41.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.563+0000 7ffb53410700 1 -- 172.21.15.78:0/3876327572 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb4c0c4540 msgr2=0x7ffb4c0c02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:41.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.563+0000 7ffb53410700 1 --2- 172.21.15.78:0/3876327572 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb4c0c4540 0x7ffb4c0c02e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c06e270 tx=0x7ffb48009bd0 comp rx=0 tx=0).stop 2023-10-14T11:58:41.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.563+0000 7ffb53410700 1 -- 172.21.15.78:0/3876327572 shutdown_connections 2023-10-14T11:58:41.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.563+0000 7ffb53410700 1 --2- 172.21.15.78:0/3876327572 >> v2:172.21.15.195:3300/0 conn(0x7ffb4c06ae10 0x7ffb4c0c0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:41.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.563+0000 7ffb53410700 1 --2- 172.21.15.78:0/3876327572 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffb4c13dbe0 0x7ffb4c06a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:41.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.563+0000 7ffb53410700 1 --2- 172.21.15.78:0/3876327572 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb4c0c4540 0x7ffb4c0c02e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:41.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.563+0000 7ffb53410700 1 -- 172.21.15.78:0/3876327572 >> 172.21.15.78:0/3876327572 conn(0x7ffb4c0c3a50 msgr2=0x7ffb4c0c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:41.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.563+0000 7ffb53410700 1 -- 172.21.15.78:0/3876327572 shutdown_connections 2023-10-14T11:58:41.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.563+0000 7ffb53410700 1 -- 172.21.15.78:0/3876327572 wait complete. 2023-10-14T11:58:41.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.564+0000 7ffb53410700 1 Processor -- start 2023-10-14T11:58:41.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.564+0000 7ffb53410700 1 -- start start 2023-10-14T11:58:41.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.565+0000 7ffb53410700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb4c06ae10 0x7ffb4c13a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:41.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.565+0000 7ffb53410700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffb4c0c4540 0x7ffb4c0ab4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:41.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.565+0000 7ffb53410700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffb4c13dbe0 0x7ffb4c0aba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:41.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.565+0000 7ffb53410700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c1edb50 con 0x7ffb4c06ae10 2023-10-14T11:58:41.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.565+0000 7ffb53410700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c0c6b90 con 0x7ffb4c13dbe0 2023-10-14T11:58:41.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.565+0000 7ffb53410700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c0abf40 con 0x7ffb4c0c4540 2023-10-14T11:58:41.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.565+0000 7ffb511ac700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb4c06ae10 0x7ffb4c13a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:41.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.565+0000 7ffb511ac700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb4c06ae10 0x7ffb4c13a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45562/0 (socket says 172.21.15.78:45562) 2023-10-14T11:58:41.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.565+0000 7ffb511ac700 1 -- 172.21.15.78:0/1948383985 learned_addr learned my addr 172.21.15.78:0/1948383985 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:41.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.565+0000 7ffb509ab700 1 --2- 172.21.15.78:0/1948383985 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffb4c0c4540 0x7ffb4c0ab4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:41.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.565+0000 7ffb519ad700 1 --2- 172.21.15.78:0/1948383985 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffb4c13dbe0 0x7ffb4c0aba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:41.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.565+0000 7ffb511ac700 1 -- 172.21.15.78:0/1948383985 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffb4c0c4540 msgr2=0x7ffb4c0ab4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:41.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.565+0000 7ffb511ac700 1 --2- 172.21.15.78:0/1948383985 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffb4c0c4540 0x7ffb4c0ab4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:41.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.565+0000 7ffb511ac700 1 -- 172.21.15.78:0/1948383985 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffb4c13dbe0 msgr2=0x7ffb4c0aba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:41.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.565+0000 7ffb511ac700 1 --2- 172.21.15.78:0/1948383985 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffb4c13dbe0 0x7ffb4c0aba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:41.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.565+0000 7ffb511ac700 1 -- 172.21.15.78:0/1948383985 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb48009550 con 0x7ffb4c06ae10 2023-10-14T11:58:41.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.565+0000 7ffb519ad700 1 --2- 172.21.15.78:0/1948383985 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffb4c13dbe0 0x7ffb4c0aba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:41.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.566+0000 7ffb511ac700 1 --2- 172.21.15.78:0/1948383985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb4c06ae10 0x7ffb4c13a490 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c00e810 tx=0x7ffb3c00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:41.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.566+0000 7ffb41ffb700 1 -- 172.21.15.78:0/1948383985 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c013610 con 0x7ffb4c06ae10 2023-10-14T11:58:41.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.566+0000 7ffb41ffb700 1 -- 172.21.15.78:0/1948383985 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb3c013770 con 0x7ffb4c06ae10 2023-10-14T11:58:41.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.566+0000 7ffb53410700 1 -- 172.21.15.78:0/1948383985 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb4c151700 con 0x7ffb4c06ae10 2023-10-14T11:58:41.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.566+0000 7ffb41ffb700 1 -- 172.21.15.78:0/1948383985 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c013b20 con 0x7ffb4c06ae10 2023-10-14T11:58:41.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.566+0000 7ffb53410700 1 -- 172.21.15.78:0/1948383985 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb4c1edd30 con 0x7ffb4c06ae10 2023-10-14T11:58:41.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.567+0000 7ffb53410700 1 -- 172.21.15.78:0/1948383985 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb4c06f560 con 0x7ffb4c06ae10 2023-10-14T11:58:41.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.567+0000 7ffb41ffb700 1 -- 172.21.15.78:0/1948383985 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ffb3c013c80 con 0x7ffb4c06ae10 2023-10-14T11:58:41.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.568+0000 7ffb41ffb700 1 --2- 172.21.15.78:0/1948383985 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ffb3006af00 0x7ffb3006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:41.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.568+0000 7ffb41ffb700 1 -- 172.21.15.78:0/1948383985 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffb3c089120 con 0x7ffb4c06ae10 2023-10-14T11:58:41.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.568+0000 7ffb509ab700 1 --2- 172.21.15.78:0/1948383985 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ffb3006af00 0x7ffb3006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:41.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.568+0000 7ffb509ab700 1 --2- 172.21.15.78:0/1948383985 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ffb3006af00 0x7ffb3006d3c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c0c4980 tx=0x7ffb48008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:41.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.572+0000 7ffb41ffb700 1 -- 172.21.15.78:0/1948383985 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ffb3c0541e0 con 0x7ffb4c06ae10 2023-10-14T11:58:41.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.759+0000 7ffb53410700 1 -- 172.21.15.78:0/1948383985 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ffb4c155090 con 0x7ffb4c06ae10 2023-10-14T11:58:41.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.759+0000 7ffb41ffb700 1 -- 172.21.15.78:0/1948383985 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7ffb3c057800 con 0x7ffb4c06ae10 2023-10-14T11:58:41.761 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2023-10-14T11:58:41.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.761+0000 7ffb53410700 1 -- 172.21.15.78:0/1948383985 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ffb3006af00 msgr2=0x7ffb3006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:41.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.761+0000 7ffb53410700 1 --2- 172.21.15.78:0/1948383985 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ffb3006af00 0x7ffb3006d3c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c0c4980 tx=0x7ffb48008030 comp rx=0 tx=0).stop 2023-10-14T11:58:41.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.761+0000 7ffb53410700 1 -- 172.21.15.78:0/1948383985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb4c06ae10 msgr2=0x7ffb4c13a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:41.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.761+0000 7ffb53410700 1 --2- 172.21.15.78:0/1948383985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb4c06ae10 0x7ffb4c13a490 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c00e810 tx=0x7ffb3c00eb20 comp rx=0 tx=0).stop 2023-10-14T11:58:41.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.761+0000 7ffb53410700 1 -- 172.21.15.78:0/1948383985 shutdown_connections 2023-10-14T11:58:41.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.761+0000 7ffb53410700 1 --2- 172.21.15.78:0/1948383985 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ffb3006af00 0x7ffb3006d3c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:41.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.761+0000 7ffb53410700 1 --2- 172.21.15.78:0/1948383985 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffb4c13dbe0 0x7ffb4c0aba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:41.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.761+0000 7ffb53410700 1 --2- 172.21.15.78:0/1948383985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb4c06ae10 0x7ffb4c13a490 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:41.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.761+0000 7ffb53410700 1 --2- 172.21.15.78:0/1948383985 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffb4c0c4540 0x7ffb4c0ab4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:41.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.761+0000 7ffb53410700 1 -- 172.21.15.78:0/1948383985 >> 172.21.15.78:0/1948383985 conn(0x7ffb4c0c3a50 msgr2=0x7ffb4c0c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:41.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.761+0000 7ffb53410700 1 -- 172.21.15.78:0/1948383985 shutdown_connections 2023-10-14T11:58:41.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.761+0000 7ffb53410700 1 -- 172.21.15.78:0/1948383985 wait complete. 2023-10-14T11:58:41.775 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop' 2023-10-14T11:58:41.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.893+0000 7fd0fcd20700 1 Processor -- start 2023-10-14T11:58:41.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.893+0000 7fd0fcd20700 1 -- start start 2023-10-14T11:58:41.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.894+0000 7fd0fcd20700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd0f80714f0 0x7fd0f80c2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:41.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.894+0000 7fd0fcd20700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0f80c1fd0 con 0x7fd0f80c2da0 2023-10-14T11:58:41.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.894+0000 7fd0fcd20700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0f80a9780 con 0x7fd0f806ab40 2023-10-14T11:58:41.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.894+0000 7fd0fcd20700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f80c1a50 con 0x7fd0f80714f0 2023-10-14T11:58:41.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.894+0000 7fd0f6d9d700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd0f80c2da0 0x7fd0f80ab3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50450/0 (socket says 172.21.15.78:50450) 2023-10-14T11:58:41.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.894+0000 7fd0f6d9d700 1 -- 172.21.15.78:0/3825935999 learned_addr learned my addr 172.21.15.78:0/3825935999 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:41.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.894+0000 7fd0f5d9b700 1 --2- 172.21.15.78:0/3825935999 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd0f80714f0 0x7fd0f80c2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:41.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.895+0000 7fd0f559a700 1 -- 172.21.15.78:0/3825935999 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 662473584 0 0) 0x7fd0f80c1fd0 con 0x7fd0f80c2da0 2023-10-14T11:58:41.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.895+0000 7fd0f559a700 1 -- 172.21.15.78:0/3825935999 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0e00035b0 con 0x7fd0f80c2da0 2023-10-14T11:58:41.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.895+0000 7fd0f559a700 1 -- 172.21.15.78:0/3825935999 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1642228982 0 0) 0x7fd0e00035b0 con 0x7fd0f80c2da0 2023-10-14T11:58:41.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.895+0000 7fd0f559a700 1 -- 172.21.15.78:0/3825935999 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd0f80714f0 msgr2=0x7fd0f80c2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:41.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.895+0000 7fd0f559a700 1 --2- 172.21.15.78:0/3825935999 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd0f80714f0 0x7fd0f80c2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:41.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.895+0000 7fd0f559a700 1 -- 172.21.15.78:0/3825935999 >> v1:172.21.15.195:6789/0 conn(0x7fd0f806ab40 legacy=0x7fd0f80aa6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:41.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.895+0000 7fd0f559a700 1 -- 172.21.15.78:0/3825935999 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f80c1340 con 0x7fd0f80c2da0 2023-10-14T11:58:41.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.895+0000 7fd0f559a700 1 -- 172.21.15.78:0/3825935999 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441799917 0 0) 0x7fd0f00019f0 con 0x7fd0f80c2da0 2023-10-14T11:58:41.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.895+0000 7fd0f559a700 1 -- 172.21.15.78:0/3825935999 >> v1:172.21.15.78:6789/0 conn(0x7fd0f80c2da0 legacy=0x7fd0f80ab3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:41.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.895+0000 7fd0f559a700 1 --2- 172.21.15.78:0/3825935999 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd0e0003a30 0x7fd0e0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:41.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.895+0000 7fd0f559a700 1 --2- 172.21.15.78:0/3825935999 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd0f80714f0 0x7fd0e0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:41.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.895+0000 7fd0f559a700 1 --2- 172.21.15.78:0/3825935999 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0e0004d90 0x7fd0e0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:41.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.895+0000 7fd0f559a700 1 -- 172.21.15.78:0/3825935999 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f80c1a50 con 0x7fd0e0004d90 2023-10-14T11:58:41.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.895+0000 7fd0f559a700 1 -- 172.21.15.78:0/3825935999 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e000bba0 con 0x7fd0e0003a30 2023-10-14T11:58:41.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.895+0000 7fd0f559a700 1 -- 172.21.15.78:0/3825935999 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e000df40 con 0x7fd0f80714f0 2023-10-14T11:58:41.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.895+0000 7fd0f6d9d700 1 --2- 172.21.15.78:0/3825935999 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0e0004d90 0x7fd0e0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:41.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.895+0000 7fd0f5d9b700 1 --2- 172.21.15.78:0/3825935999 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd0f80714f0 0x7fd0e0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:41.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.895+0000 7fd0f659c700 1 --2- 172.21.15.78:0/3825935999 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd0e0003a30 0x7fd0e0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:41.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.895+0000 7fd0f6d9d700 1 -- 172.21.15.78:0/3825935999 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd0f80714f0 msgr2=0x7fd0e0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:41.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.895+0000 7fd0f6d9d700 1 --2- 172.21.15.78:0/3825935999 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd0f80714f0 0x7fd0e0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:41.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.895+0000 7fd0f6d9d700 1 -- 172.21.15.78:0/3825935999 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd0e0003a30 msgr2=0x7fd0e0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:41.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.895+0000 7fd0f6d9d700 1 --2- 172.21.15.78:0/3825935999 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd0e0003a30 0x7fd0e0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:41.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.895+0000 7fd0f6d9d700 1 -- 172.21.15.78:0/3825935999 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e00035b0 con 0x7fd0e0004d90 2023-10-14T11:58:41.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.895+0000 7fd0f6d9d700 1 --2- 172.21.15.78:0/3825935999 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0e0004d90 0x7fd0e0005210 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd0f000a860 tx=0x7fd0e0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cbbdd4b908e58c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:41.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.896+0000 7fd0f559a700 1 -- 172.21.15.78:0/3825935999 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f0012600 con 0x7fd0e0004d90 2023-10-14T11:58:41.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.896+0000 7fd0f559a700 1 -- 172.21.15.78:0/3825935999 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0f0012780 con 0x7fd0e0004d90 2023-10-14T11:58:41.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.896+0000 7fd0f559a700 1 -- 172.21.15.78:0/3825935999 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f0012b30 con 0x7fd0e0004d90 2023-10-14T11:58:41.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.896+0000 7fd0fcd20700 1 -- 172.21.15.78:0/3825935999 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0e0004d90 msgr2=0x7fd0e0005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:41.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.896+0000 7fd0fcd20700 1 --2- 172.21.15.78:0/3825935999 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0e0004d90 0x7fd0e0005210 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd0f000a860 tx=0x7fd0e0005880 comp rx=0 tx=0).stop 2023-10-14T11:58:41.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.896+0000 7fd0f6d9d700 1 -- 172.21.15.78:0/3825935999 reap_dead start 2023-10-14T11:58:41.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.896+0000 7fd0fcd20700 1 -- 172.21.15.78:0/3825935999 shutdown_connections 2023-10-14T11:58:41.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.896+0000 7fd0fcd20700 1 -- 172.21.15.78:0/3825935999 >> 172.21.15.78:0/3825935999 conn(0x7fd0f80a8410 msgr2=0x7fd0f80a8820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:41.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.896+0000 7fd0fcd20700 1 -- 172.21.15.78:0/3825935999 shutdown_connections 2023-10-14T11:58:41.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.896+0000 7fd0fcd20700 1 -- 172.21.15.78:0/3825935999 wait complete. 2023-10-14T11:58:41.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.897+0000 7fd0fcd20700 1 Processor -- start 2023-10-14T11:58:41.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.898+0000 7fd0fcd20700 1 -- start start 2023-10-14T11:58:41.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.898+0000 7fd0fcd20700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd0e0004d90 0x7fd0f81da530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:41.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.898+0000 7fd0fcd20700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd0f81d7a20 0x7fd0f80be4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:41.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.898+0000 7fd0fcd20700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0f81ed0c0 0x7fd0f81ed540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:41.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.898+0000 7fd0fcd20700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f81daa70 con 0x7fd0f81ed0c0 2023-10-14T11:58:41.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.898+0000 7fd0fcd20700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f81d94d0 con 0x7fd0e0004d90 2023-10-14T11:58:41.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.898+0000 7fd0fcd20700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f81d9280 con 0x7fd0f81d7a20 2023-10-14T11:58:41.919 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.898+0000 7fd0f6d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0f81ed0c0 0x7fd0f81ed540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:41.919 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.898+0000 7fd0f6d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0f81ed0c0 0x7fd0f81ed540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45586/0 (socket says 172.21.15.78:45586) 2023-10-14T11:58:41.919 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.898+0000 7fd0f6d9d700 1 -- 172.21.15.78:0/2051180522 learned_addr learned my addr 172.21.15.78:0/2051180522 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:41.919 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.898+0000 7fd0f5d9b700 1 --2- 172.21.15.78:0/2051180522 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd0f81d7a20 0x7fd0f80be4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:41.919 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.898+0000 7fd0f659c700 1 --2- 172.21.15.78:0/2051180522 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd0e0004d90 0x7fd0f81da530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:41.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.899+0000 7fd0f6d9d700 1 -- 172.21.15.78:0/2051180522 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd0f81d7a20 msgr2=0x7fd0f80be4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:41.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.899+0000 7fd0f6d9d700 1 --2- 172.21.15.78:0/2051180522 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd0f81d7a20 0x7fd0f80be4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:41.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.899+0000 7fd0f6d9d700 1 -- 172.21.15.78:0/2051180522 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd0e0004d90 msgr2=0x7fd0f81da530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:41.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.899+0000 7fd0f6d9d700 1 --2- 172.21.15.78:0/2051180522 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd0e0004d90 0x7fd0f81da530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:41.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.899+0000 7fd0f6d9d700 1 -- 172.21.15.78:0/2051180522 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f81d16d0 con 0x7fd0f81ed0c0 2023-10-14T11:58:41.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.899+0000 7fd0f6d9d700 1 --2- 172.21.15.78:0/2051180522 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0f81ed0c0 0x7fd0f81ed540 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd0e00057f0 tx=0x7fd0f0001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:41.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.899+0000 7fd0e6ffd700 1 -- 172.21.15.78:0/2051180522 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f00019b0 con 0x7fd0f81ed0c0 2023-10-14T11:58:41.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.899+0000 7fd0e6ffd700 1 -- 172.21.15.78:0/2051180522 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0f0012610 con 0x7fd0f81ed0c0 2023-10-14T11:58:41.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.899+0000 7fd0fcd20700 1 -- 172.21.15.78:0/2051180522 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0f81da050 con 0x7fd0f81ed0c0 2023-10-14T11:58:41.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.899+0000 7fd0e6ffd700 1 -- 172.21.15.78:0/2051180522 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f001e630 con 0x7fd0f81ed0c0 2023-10-14T11:58:41.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.899+0000 7fd0fcd20700 1 -- 172.21.15.78:0/2051180522 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0f81d1360 con 0x7fd0f81ed0c0 2023-10-14T11:58:41.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.900+0000 7fd0fcd20700 1 -- 172.21.15.78:0/2051180522 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0f81d1860 con 0x7fd0f81ed0c0 2023-10-14T11:58:41.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.900+0000 7fd0e6ffd700 1 -- 172.21.15.78:0/2051180522 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd0f001e790 con 0x7fd0f81ed0c0 2023-10-14T11:58:41.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.900+0000 7fd0e6ffd700 1 --2- 172.21.15.78:0/2051180522 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd0d806af00 0x7fd0d806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:41.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.901+0000 7fd0e6ffd700 1 -- 172.21.15.78:0/2051180522 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd0f0023020 con 0x7fd0f81ed0c0 2023-10-14T11:58:41.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.901+0000 7fd0f659c700 1 --2- 172.21.15.78:0/2051180522 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd0d806af00 0x7fd0d806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:41.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.901+0000 7fd0f659c700 1 --2- 172.21.15.78:0/2051180522 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd0d806af00 0x7fd0d806d3c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec000c00 tx=0x7fd0ec009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:41.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:41.905+0000 7fd0e6ffd700 1 -- 172.21.15.78:0/2051180522 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd0f00571f0 con 0x7fd0f81ed0c0 2023-10-14T11:58:42.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.172+0000 7fd0fcd20700 1 -- 172.21.15.78:0/2051180522 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop"]} v 0) v1 -- 0x7fd0f8074b80 con 0x7fd0f81ed0c0 2023-10-14T11:58:42.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.701+0000 7fd0e6ffd700 1 -- 172.21.15.78:0/2051180522 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop"]}]=0 v43) v1 ==== 146+0+0 (secure 0 0 0) 0x7fd0f005a810 con 0x7fd0f81ed0c0 2023-10-14T11:58:42.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.704+0000 7fd0fcd20700 1 -- 172.21.15.78:0/2051180522 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd0d806af00 msgr2=0x7fd0d806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:42.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.704+0000 7fd0fcd20700 1 --2- 172.21.15.78:0/2051180522 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd0d806af00 0x7fd0d806d3c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec000c00 tx=0x7fd0ec009040 comp rx=0 tx=0).stop 2023-10-14T11:58:42.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.704+0000 7fd0fcd20700 1 -- 172.21.15.78:0/2051180522 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0f81ed0c0 msgr2=0x7fd0f81ed540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:42.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.704+0000 7fd0fcd20700 1 --2- 172.21.15.78:0/2051180522 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0f81ed0c0 0x7fd0f81ed540 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd0e00057f0 tx=0x7fd0f0001d80 comp rx=0 tx=0).stop 2023-10-14T11:58:42.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.704+0000 7fd0fcd20700 1 -- 172.21.15.78:0/2051180522 shutdown_connections 2023-10-14T11:58:42.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.704+0000 7fd0fcd20700 1 --2- 172.21.15.78:0/2051180522 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0f81ed0c0 0x7fd0f81ed540 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:42.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.704+0000 7fd0fcd20700 1 --2- 172.21.15.78:0/2051180522 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd0d806af00 0x7fd0d806d3c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:42.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.704+0000 7fd0fcd20700 1 --2- 172.21.15.78:0/2051180522 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd0e0004d90 0x7fd0f81da530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:42.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.704+0000 7fd0fcd20700 1 --2- 172.21.15.78:0/2051180522 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd0f81d7a20 0x7fd0f80be4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:42.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.704+0000 7fd0fcd20700 1 -- 172.21.15.78:0/2051180522 >> 172.21.15.78:0/2051180522 conn(0x7fd0f80a8410 msgr2=0x7fd0f80c4d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:42.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.704+0000 7fd0fcd20700 1 -- 172.21.15.78:0/2051180522 shutdown_connections 2023-10-14T11:58:42.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.704+0000 7fd0fcd20700 1 -- 172.21.15.78:0/2051180522 wait complete. 2023-10-14T11:58:42.720 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-14T11:58:42.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.846+0000 7f3c179f9700 1 Processor -- start 2023-10-14T11:58:42.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.846+0000 7f3c179f9700 1 -- start start 2023-10-14T11:58:42.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.847+0000 7f3c179f9700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3c100c4540 0x7f3c100c02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:42.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.847+0000 7f3c179f9700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c1006b500 0x7f3c100c0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:42.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.847+0000 7f3c179f9700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c1006ebe0 con 0x7f3c1013dbe0 2023-10-14T11:58:42.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.847+0000 7f3c179f9700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c1006d330 con 0x7f3c100c4540 2023-10-14T11:58:42.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.847+0000 7f3c179f9700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c100c0d60 con 0x7f3c1006b500 2023-10-14T11:58:42.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.847+0000 7f3c14f94700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3c100c4540 0x7f3c100c02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:42.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.847+0000 7f3c15795700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f3c1013dbe0 0x7f3c1006a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50466/0 (socket says 172.21.15.78:50466) 2023-10-14T11:58:42.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.847+0000 7f3c15795700 1 -- 172.21.15.78:0/1405310512 learned_addr learned my addr 172.21.15.78:0/1405310512 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:42.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.848+0000 7f3c15f96700 1 --2- 172.21.15.78:0/1405310512 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c1006b500 0x7f3c100c0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:42.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.848+0000 7f3c14f94700 1 -- 172.21.15.78:0/1405310512 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c1006b500 msgr2=0x7f3c100c0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:42.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.848+0000 7f3c14f94700 1 --2- 172.21.15.78:0/1405310512 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c1006b500 0x7f3c100c0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:42.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.848+0000 7f3c14f94700 1 -- 172.21.15.78:0/1405310512 >> v1:172.21.15.78:6789/0 conn(0x7f3c1013dbe0 legacy=0x7f3c1006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:42.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.848+0000 7f3c14f94700 1 -- 172.21.15.78:0/1405310512 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c101558c0 con 0x7f3c100c4540 2023-10-14T11:58:42.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.848+0000 7f3c03fff700 1 -- 172.21.15.78:0/1405310512 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3317481059 0 0) 0x7f3c1006ebe0 con 0x7f3c1013dbe0 2023-10-14T11:58:42.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.848+0000 7f3c14f94700 1 --2- 172.21.15.78:0/1405310512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c100c4540 0x7f3c100c02e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c009840 tx=0x7f3c0c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a7f8d97ae54753e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:42.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.848+0000 7f3c03fff700 1 -- 172.21.15.78:0/1405310512 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0c011830 con 0x7f3c100c4540 2023-10-14T11:58:42.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.848+0000 7f3c03fff700 1 -- 172.21.15.78:0/1405310512 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c0c011990 con 0x7f3c100c4540 2023-10-14T11:58:42.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.848+0000 7f3c03fff700 1 -- 172.21.15.78:0/1405310512 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0c011d20 con 0x7f3c100c4540 2023-10-14T11:58:42.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.849+0000 7f3c179f9700 1 -- 172.21.15.78:0/1405310512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c100c4540 msgr2=0x7f3c100c02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:42.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.849+0000 7f3c179f9700 1 --2- 172.21.15.78:0/1405310512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c100c4540 0x7f3c100c02e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c009840 tx=0x7f3c0c009b50 comp rx=0 tx=0).stop 2023-10-14T11:58:42.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.849+0000 7f3c179f9700 1 -- 172.21.15.78:0/1405310512 shutdown_connections 2023-10-14T11:58:42.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.849+0000 7f3c179f9700 1 --2- 172.21.15.78:0/1405310512 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c1006b500 0x7f3c100c0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:42.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.849+0000 7f3c179f9700 1 --2- 172.21.15.78:0/1405310512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c100c4540 0x7f3c100c02e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:42.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.849+0000 7f3c179f9700 1 -- 172.21.15.78:0/1405310512 >> 172.21.15.78:0/1405310512 conn(0x7f3c100c3a50 msgr2=0x7f3c100c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:42.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.849+0000 7f3c179f9700 1 -- 172.21.15.78:0/1405310512 shutdown_connections 2023-10-14T11:58:42.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.849+0000 7f3c179f9700 1 -- 172.21.15.78:0/1405310512 wait complete. 2023-10-14T11:58:42.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.850+0000 7f3c179f9700 1 Processor -- start 2023-10-14T11:58:42.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.850+0000 7f3c179f9700 1 -- start start 2023-10-14T11:58:42.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.851+0000 7f3c179f9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c1006b500 0x7f3c100c6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:42.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.851+0000 7f3c179f9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c100c4540 0x7f3c101e9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:42.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.851+0000 7f3c179f9700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c1013dbe0 0x7f3c101e9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:42.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.851+0000 7f3c179f9700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c100ac520 con 0x7f3c1006b500 2023-10-14T11:58:42.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.851+0000 7f3c179f9700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c101ed460 con 0x7f3c100c4540 2023-10-14T11:58:42.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.851+0000 7f3c179f9700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c100abf60 con 0x7f3c1013dbe0 2023-10-14T11:58:42.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.851+0000 7f3c15795700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c1006b500 0x7f3c100c6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:42.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.851+0000 7f3c15795700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c1006b500 0x7f3c100c6090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45596/0 (socket says 172.21.15.78:45596) 2023-10-14T11:58:42.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.851+0000 7f3c15795700 1 -- 172.21.15.78:0/818100906 learned_addr learned my addr 172.21.15.78:0/818100906 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:42.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.851+0000 7f3c15795700 1 -- 172.21.15.78:0/818100906 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c1013dbe0 msgr2=0x7f3c101e9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:42.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.851+0000 7f3c15795700 1 --2- 172.21.15.78:0/818100906 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c1013dbe0 0x7f3c101e9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:42.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.851+0000 7f3c15795700 1 -- 172.21.15.78:0/818100906 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c100c4540 msgr2=0x7f3c101e9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:42.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.851+0000 7f3c15795700 1 --2- 172.21.15.78:0/818100906 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c100c4540 0x7f3c101e9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:42.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.851+0000 7f3c15795700 1 -- 172.21.15.78:0/818100906 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0c0094a0 con 0x7f3c1006b500 2023-10-14T11:58:42.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.852+0000 7f3c15795700 1 --2- 172.21.15.78:0/818100906 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c1006b500 0x7f3c100c6090 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3c040011e0 tx=0x7f3c0400cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:42.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.852+0000 7f3c01ffb700 1 -- 172.21.15.78:0/818100906 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c04004c80 con 0x7f3c1006b500 2023-10-14T11:58:42.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.852+0000 7f3c01ffb700 1 -- 172.21.15.78:0/818100906 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c04004e00 con 0x7f3c1006b500 2023-10-14T11:58:42.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.852+0000 7f3c179f9700 1 -- 172.21.15.78:0/818100906 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c100ac240 con 0x7f3c1006b500 2023-10-14T11:58:42.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.852+0000 7f3c01ffb700 1 -- 172.21.15.78:0/818100906 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c04007310 con 0x7f3c1006b500 2023-10-14T11:58:42.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.852+0000 7f3c179f9700 1 -- 172.21.15.78:0/818100906 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c101ed080 con 0x7f3c1006b500 2023-10-14T11:58:42.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.853+0000 7f3c179f9700 1 -- 172.21.15.78:0/818100906 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c10157dd0 con 0x7f3c1006b500 2023-10-14T11:58:42.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.853+0000 7f3c01ffb700 1 -- 172.21.15.78:0/818100906 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3c04007470 con 0x7f3c1006b500 2023-10-14T11:58:42.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.854+0000 7f3c01ffb700 1 --2- 172.21.15.78:0/818100906 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f3bf406af00 0x7f3bf406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:42.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.854+0000 7f3c01ffb700 1 -- 172.21.15.78:0/818100906 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3c04089410 con 0x7f3c1006b500 2023-10-14T11:58:42.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.854+0000 7f3c14f94700 1 --2- 172.21.15.78:0/818100906 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f3bf406af00 0x7f3bf406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:42.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.854+0000 7f3c14f94700 1 --2- 172.21.15.78:0/818100906 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f3bf406af00 0x7f3bf406d3c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3c100c4980 tx=0x7f3c0c010000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:42.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:42.858+0000 7f3c01ffb700 1 -- 172.21.15.78:0/818100906 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3c040544c0 con 0x7f3c1006b500 2023-10-14T11:58:43.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.046+0000 7f3c179f9700 1 -- 172.21.15.78:0/818100906 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f3c10074b80 con 0x7f3c1006b500 2023-10-14T11:58:43.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.051+0000 7f3c01ffb700 1 -- 172.21.15.78:0/818100906 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v11)=0 v11) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3c04057ae0 con 0x7f3c1006b500 2023-10-14T11:58:43.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.053+0000 7f3c179f9700 1 -- 172.21.15.78:0/818100906 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f3bf406af00 msgr2=0x7f3bf406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:43.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.053+0000 7f3c179f9700 1 --2- 172.21.15.78:0/818100906 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f3bf406af00 0x7f3bf406d3c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3c100c4980 tx=0x7f3c0c010000 comp rx=0 tx=0).stop 2023-10-14T11:58:43.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.053+0000 7f3c179f9700 1 -- 172.21.15.78:0/818100906 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c1006b500 msgr2=0x7f3c100c6090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:43.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.053+0000 7f3c179f9700 1 --2- 172.21.15.78:0/818100906 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c1006b500 0x7f3c100c6090 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3c040011e0 tx=0x7f3c0400cae0 comp rx=0 tx=0).stop 2023-10-14T11:58:43.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.053+0000 7f3c179f9700 1 -- 172.21.15.78:0/818100906 shutdown_connections 2023-10-14T11:58:43.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.053+0000 7f3c179f9700 1 --2- 172.21.15.78:0/818100906 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c1013dbe0 0x7f3c101e9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:43.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.053+0000 7f3c179f9700 1 --2- 172.21.15.78:0/818100906 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c1006b500 0x7f3c100c6090 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:43.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.053+0000 7f3c179f9700 1 --2- 172.21.15.78:0/818100906 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f3bf406af00 0x7f3bf406d3c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:43.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.053+0000 7f3c179f9700 1 --2- 172.21.15.78:0/818100906 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c100c4540 0x7f3c101e9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:43.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.053+0000 7f3c179f9700 1 -- 172.21.15.78:0/818100906 >> 172.21.15.78:0/818100906 conn(0x7f3c100c3a50 msgr2=0x7f3c100c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:43.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.053+0000 7f3c179f9700 1 -- 172.21.15.78:0/818100906 shutdown_connections 2023-10-14T11:58:43.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.053+0000 7f3c179f9700 1 -- 172.21.15.78:0/818100906 wait complete. 2023-10-14T11:58:43.067 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-14T11:58:43.068 DEBUG:teuthology.orchestra.run.smithi078:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T11:58:43.108 DEBUG:teuthology.orchestra.run.smithi195:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T11:58:43.153 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-14T11:58:43.154 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-14T11:58:43.154 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T11:58:43.171 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-14T11:58:43.172 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T11:58:43.229 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-14T11:58:43.229 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T11:58:43.286 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T11:58:43.444 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.442+0000 7f8f68e89700 1 Processor -- start 2023-10-14T11:58:43.445 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.442+0000 7f8f68e89700 1 -- start start 2023-10-14T11:58:43.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.443+0000 7f8f68e89700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8f6414b6b0 0x7f8f64072580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:43.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.443+0000 7f8f68e89700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f8f6406db50 0x7f8f640c2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:43.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.443+0000 7f8f68e89700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f640c25e0 con 0x7f8f640c2fb0 2023-10-14T11:58:43.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.443+0000 7f8f68e89700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f640a87b0 con 0x7f8f6414b6b0 2023-10-14T11:58:43.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.443+0000 7f8f68e89700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f6414ba90 con 0x7f8f6406db50 2023-10-14T11:58:43.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.444+0000 7f8f6259c700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8f6414b6b0 0x7f8f64072580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:43.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.444+0000 7f8f6259c700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8f6414b6b0 0x7f8f64072580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45604/0 (socket says 172.21.15.78:45604) 2023-10-14T11:58:43.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.444+0000 7f8f6259c700 1 -- 172.21.15.78:0/1737075331 learned_addr learned my addr 172.21.15.78:0/1737075331 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:43.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.444+0000 7f8f61d9b700 1 --2- 172.21.15.78:0/1737075331 >> v2:172.21.15.195:3300/0 conn(0x7f8f6406db50 0x7f8f640c2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:43.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.444+0000 7f8f6259c700 1 -- 172.21.15.78:0/1737075331 >> v2:172.21.15.195:3300/0 conn(0x7f8f6406db50 msgr2=0x7f8f640c2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:43.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.444+0000 7f8f6259c700 1 --2- 172.21.15.78:0/1737075331 >> v2:172.21.15.195:3300/0 conn(0x7f8f6406db50 0x7f8f640c2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:43.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.444+0000 7f8f6259c700 1 -- 172.21.15.78:0/1737075331 >> v1:172.21.15.195:6789/0 conn(0x7f8f640c2fb0 legacy=0x7f8f64154bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:43.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.444+0000 7f8f6259c700 1 -- 172.21.15.78:0/1737075331 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f64155660 con 0x7f8f6414b6b0 2023-10-14T11:58:43.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.444+0000 7f8f61d9b700 1 --2- 172.21.15.78:0/1737075331 >> v2:172.21.15.195:3300/0 conn(0x7f8f6406db50 0x7f8f640c2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:43.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.444+0000 7f8f6259c700 1 --2- 172.21.15.78:0/1737075331 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f6414b6b0 0x7f8f64072580 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8f640c1890 tx=0x7f8f54009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9fc17f72326148b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:43.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.444+0000 7f8f6159a700 1 -- 172.21.15.78:0/1737075331 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f54004230 con 0x7f8f6414b6b0 2023-10-14T11:58:43.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.444+0000 7f8f6159a700 1 -- 172.21.15.78:0/1737075331 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f54004390 con 0x7f8f6414b6b0 2023-10-14T11:58:43.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.444+0000 7f8f6159a700 1 -- 172.21.15.78:0/1737075331 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f540036a0 con 0x7f8f6414b6b0 2023-10-14T11:58:43.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.445+0000 7f8f68e89700 1 -- 172.21.15.78:0/1737075331 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f6414b6b0 msgr2=0x7f8f64072580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:43.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.445+0000 7f8f68e89700 1 --2- 172.21.15.78:0/1737075331 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f6414b6b0 0x7f8f64072580 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8f640c1890 tx=0x7f8f54009bd0 comp rx=0 tx=0).stop 2023-10-14T11:58:43.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.445+0000 7f8f68e89700 1 -- 172.21.15.78:0/1737075331 shutdown_connections 2023-10-14T11:58:43.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.445+0000 7f8f68e89700 1 --2- 172.21.15.78:0/1737075331 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f6414b6b0 0x7f8f64072580 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:43.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.445+0000 7f8f68e89700 1 --2- 172.21.15.78:0/1737075331 >> v2:172.21.15.195:3300/0 conn(0x7f8f6406db50 0x7f8f640c2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:43.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.445+0000 7f8f68e89700 1 -- 172.21.15.78:0/1737075331 >> 172.21.15.78:0/1737075331 conn(0x7f8f640c3ac0 msgr2=0x7f8f640c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:43.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.445+0000 7f8f68e89700 1 -- 172.21.15.78:0/1737075331 shutdown_connections 2023-10-14T11:58:43.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.445+0000 7f8f68e89700 1 -- 172.21.15.78:0/1737075331 wait complete. 2023-10-14T11:58:43.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.446+0000 7f8f68e89700 1 Processor -- start 2023-10-14T11:58:43.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.446+0000 7f8f68e89700 1 -- start start 2023-10-14T11:58:43.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.447+0000 7f8f68e89700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f6406db50 0x7f8f641eb790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:43.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.447+0000 7f8f68e89700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8f640c2fb0 0x7f8f6406ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:43.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.447+0000 7f8f68e89700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8f6414b6b0 0x7f8f641e8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:43.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.447+0000 7f8f68e89700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f641ebcd0 con 0x7f8f6406db50 2023-10-14T11:58:43.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.447+0000 7f8f68e89700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f640c52d0 con 0x7f8f640c2fb0 2023-10-14T11:58:43.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.447+0000 7f8f68e89700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f640c55b0 con 0x7f8f6414b6b0 2023-10-14T11:58:43.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.447+0000 7f8f6259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f6406db50 0x7f8f641eb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:43.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.447+0000 7f8f6259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f6406db50 0x7f8f641eb790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45610/0 (socket says 172.21.15.78:45610) 2023-10-14T11:58:43.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.447+0000 7f8f6259c700 1 -- 172.21.15.78:0/4179061287 learned_addr learned my addr 172.21.15.78:0/4179061287 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:43.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.447+0000 7f8f61d9b700 1 --2- 172.21.15.78:0/4179061287 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8f640c2fb0 0x7f8f6406ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:43.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.447+0000 7f8f62d9d700 1 --2- 172.21.15.78:0/4179061287 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8f6414b6b0 0x7f8f641e8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:43.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.447+0000 7f8f6259c700 1 -- 172.21.15.78:0/4179061287 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8f6414b6b0 msgr2=0x7f8f641e8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:43.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.447+0000 7f8f6259c700 1 --2- 172.21.15.78:0/4179061287 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8f6414b6b0 0x7f8f641e8bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:43.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.447+0000 7f8f6259c700 1 -- 172.21.15.78:0/4179061287 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8f640c2fb0 msgr2=0x7f8f6406ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:43.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.447+0000 7f8f6259c700 1 --2- 172.21.15.78:0/4179061287 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8f640c2fb0 0x7f8f6406ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:43.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.447+0000 7f8f6259c700 1 -- 172.21.15.78:0/4179061287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f54009550 con 0x7f8f6406db50 2023-10-14T11:58:43.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.447+0000 7f8f62d9d700 1 --2- 172.21.15.78:0/4179061287 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8f6414b6b0 0x7f8f641e8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:43.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.447+0000 7f8f6259c700 1 --2- 172.21.15.78:0/4179061287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f6406db50 0x7f8f641eb790 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8f54009130 tx=0x7f8f54004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:43.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.447+0000 7f8f5affd700 1 -- 172.21.15.78:0/4179061287 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f54011630 con 0x7f8f6406db50 2023-10-14T11:58:43.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.448+0000 7f8f5affd700 1 -- 172.21.15.78:0/4179061287 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f54011790 con 0x7f8f6406db50 2023-10-14T11:58:43.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.448+0000 7f8f68e89700 1 -- 172.21.15.78:0/4179061287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f640c5890 con 0x7f8f6406db50 2023-10-14T11:58:43.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.448+0000 7f8f5affd700 1 -- 172.21.15.78:0/4179061287 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f54011b40 con 0x7f8f6406db50 2023-10-14T11:58:43.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.448+0000 7f8f68e89700 1 -- 172.21.15.78:0/4179061287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f640c5e50 con 0x7f8f6406db50 2023-10-14T11:58:43.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.448+0000 7f8f68e89700 1 -- 172.21.15.78:0/4179061287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f64074b80 con 0x7f8f6406db50 2023-10-14T11:58:43.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.449+0000 7f8f5affd700 1 -- 172.21.15.78:0/4179061287 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8f54011ca0 con 0x7f8f6406db50 2023-10-14T11:58:43.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.449+0000 7f8f5affd700 1 --2- 172.21.15.78:0/4179061287 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8f4406af00 0x7f8f4406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:43.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.449+0000 7f8f61d9b700 1 --2- 172.21.15.78:0/4179061287 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8f4406af00 0x7f8f4406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:43.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.449+0000 7f8f5affd700 1 -- 172.21.15.78:0/4179061287 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8f5408ac10 con 0x7f8f6406db50 2023-10-14T11:58:43.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.450+0000 7f8f61d9b700 1 --2- 172.21.15.78:0/4179061287 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8f4406af00 0x7f8f4406d3c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8f641418e0 tx=0x7f8f4c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:43.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.454+0000 7f8f5affd700 1 -- 172.21.15.78:0/4179061287 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8f54055d50 con 0x7f8f6406db50 2023-10-14T11:58:43.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.644+0000 7f8f68e89700 1 -- 172.21.15.78:0/4179061287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f64072360 con 0x7f8f6406db50 2023-10-14T11:58:43.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.645+0000 7f8f5affd700 1 -- 172.21.15.78:0/4179061287 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16481 (secure 0 0 0) 0x7f8f5401f090 con 0x7f8f6406db50 2023-10-14T11:58:43.647 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:43.647 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":21,"fsid":"4c5dc134-3092-406a-a1e1-13e6d9e926e8","created":"2023-10-14T11:57:57.363750+0000","modified":"2023-10-14T11:58:16.139413+0000","last_up_change":"2023-10-14T11:58:06.982472+0000","last_in_change":"2023-10-14T11:57:59.821373+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T11:58:09.001200+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T11:58:12.778573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T11:58:13.490286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"625a5fc0-9474-4eae-b80e-14c27a74ad19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6801","nonce":3884969780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6803","nonce":3884969780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6807","nonce":3884969780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6805","nonce":3884969780}]},"public_addr":"172.21.15.78:6801/3884969780","cluster_addr":"172.21.15.78:6803/3884969780","heartbeat_back_addr":"172.21.15.78:6807/3884969780","heartbeat_front_addr":"172.21.15.78:6805/3884969780","state":["exists","up"]},{"osd":1,"uuid":"49d63db1-2981-49ab-9c40-c50fc744b982","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6809","nonce":2176928406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6811","nonce":2176928406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6815","nonce":2176928406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6813","nonce":2176928406}]},"public_addr":"172.21.15.78:6809/2176928406","cluster_addr":"172.21.15.78:6811/2176928406","heartbeat_back_addr":"172.21.15.78:6815/2176928406","heartbeat_front_addr":"172.21.15.78:6813/2176928406","state":["exists","up"]},{"osd":2,"uuid":"1d7d0b29-cbd1-4717-9fe6-765a8d4998b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6825","nonce":3647044577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6827","nonce":3647044577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6831","nonce":3647044577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6829","nonce":3647044577}]},"public_addr":"172.21.15.78:6825/3647044577","cluster_addr":"172.21.15.78:6827/3647044577","heartbeat_back_addr":"172.21.15.78:6831/3647044577","heartbeat_front_addr":"172.21.15.78:6829/3647044577","state":["exists","up"]},{"osd":3,"uuid":"2f75ded1-e9e2-40e7-99ae-45e1e425e1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6817","nonce":3584112288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6819","nonce":3584112288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6823","nonce":3584112288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6821","nonce":3584112288}]},"public_addr":"172.21.15.78:6817/3584112288","cluster_addr":"172.21.15.78:6819/3584112288","heartbeat_back_addr":"172.21.15.78:6823/3584112288","heartbeat_front_addr":"172.21.15.78:6821/3584112288","state":["exists","up"]},{"osd":4,"uuid":"5130b432-8358-46cb-8a31-3064c976f6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6801","nonce":3732756301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6803","nonce":3732756301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6807","nonce":3732756301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6805","nonce":3732756301}]},"public_addr":"172.21.15.195:6801/3732756301","cluster_addr":"172.21.15.195:6803/3732756301","heartbeat_back_addr":"172.21.15.195:6807/3732756301","heartbeat_front_addr":"172.21.15.195:6805/3732756301","state":["exists","up"]},{"osd":5,"uuid":"60e234a9-a506-4e40-9c19-49a58f3cf28c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6825","nonce":2857706411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6827","nonce":2857706411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6831","nonce":2857706411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6829","nonce":2857706411}]},"public_addr":"172.21.15.195:6825/2857706411","cluster_addr":"172.21.15.195:6827/2857706411","heartbeat_back_addr":"172.21.15.195:6831/2857706411","heartbeat_front_addr":"172.21.15.195:6829/2857706411","state":["exists","up"]},{"osd":6,"uuid":"df7d2dd6-26b6-4eb1-96e5-94d1f3955b33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6810","nonce":72554374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6813","nonce":72554374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6820","nonce":72554374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6817","nonce":72554374}]},"public_addr":"172.21.15.195:6810/72554374","cluster_addr":"172.21.15.195:6813/72554374","heartbeat_back_addr":"172.21.15.195:6820/72554374","heartbeat_front_addr":"172.21.15.195:6817/72554374","state":["exists","up"]},{"osd":7,"uuid":"47a9d4de-f56d-4aa4-9900-c4a9cf9b2f97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6809","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6811","nonce":482472979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6816","nonce":482472979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6823","nonce":482472979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6821","nonce":482472979}]},"public_addr":"172.21.15.195:6811/482472979","cluster_addr":"172.21.15.195:6816/482472979","heartbeat_back_addr":"172.21.15.195:6823/482472979","heartbeat_front_addr":"172.21.15.195:6821/482472979","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.754003+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.005994+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.043603+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.024506+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.062585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.033767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.994825+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.038300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T11:58:43.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.647+0000 7f8f68e89700 1 -- 172.21.15.78:0/4179061287 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8f4406af00 msgr2=0x7f8f4406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:43.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.647+0000 7f8f68e89700 1 --2- 172.21.15.78:0/4179061287 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8f4406af00 0x7f8f4406d3c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8f641418e0 tx=0x7f8f4c008040 comp rx=0 tx=0).stop 2023-10-14T11:58:43.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.647+0000 7f8f68e89700 1 -- 172.21.15.78:0/4179061287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f6406db50 msgr2=0x7f8f641eb790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:43.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.647+0000 7f8f68e89700 1 --2- 172.21.15.78:0/4179061287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f6406db50 0x7f8f641eb790 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8f54009130 tx=0x7f8f54004750 comp rx=0 tx=0).stop 2023-10-14T11:58:43.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.647+0000 7f8f68e89700 1 -- 172.21.15.78:0/4179061287 shutdown_connections 2023-10-14T11:58:43.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.647+0000 7f8f68e89700 1 --2- 172.21.15.78:0/4179061287 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8f6414b6b0 0x7f8f641e8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:43.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.647+0000 7f8f68e89700 1 --2- 172.21.15.78:0/4179061287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f6406db50 0x7f8f641eb790 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:43.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.647+0000 7f8f68e89700 1 --2- 172.21.15.78:0/4179061287 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8f4406af00 0x7f8f4406d3c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:43.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.647+0000 7f8f68e89700 1 --2- 172.21.15.78:0/4179061287 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8f640c2fb0 0x7f8f6406ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:43.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.647+0000 7f8f68e89700 1 -- 172.21.15.78:0/4179061287 >> 172.21.15.78:0/4179061287 conn(0x7f8f640c3ac0 msgr2=0x7f8f6414a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:43.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.647+0000 7f8f68e89700 1 -- 172.21.15.78:0/4179061287 shutdown_connections 2023-10-14T11:58:43.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.647+0000 7f8f68e89700 1 -- 172.21.15.78:0/4179061287 wait complete. 2023-10-14T11:58:43.661 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T11:58:09.001200+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T11:58:12.778573+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T11:58:13.490286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T11:58:43.662 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T11:58:43.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.786+0000 7fb5631f9700 1 Processor -- start 2023-10-14T11:58:43.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.786+0000 7fb5631f9700 1 -- start start 2023-10-14T11:58:43.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.787+0000 7fb5631f9700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb55c0c11a0 0x7fb55c14dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:43.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.787+0000 7fb5631f9700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb55c1542e0 con 0x7fb55c0c69e0 2023-10-14T11:58:43.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.787+0000 7fb5631f9700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb55c1519f0 con 0x7fb55c14e520 2023-10-14T11:58:43.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.787+0000 7fb5631f9700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb55c151f20 con 0x7fb55c0c11a0 2023-10-14T11:58:43.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.787+0000 7fb55bfff700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb55c0c11a0 0x7fb55c14dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:43.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.787+0000 7fb560f95700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb55c0c69e0 0x7fb55c06e300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50494/0 (socket says 172.21.15.78:50494) 2023-10-14T11:58:43.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.787+0000 7fb55bfff700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb55c0c11a0 0x7fb55c14dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45620/0 (socket says 172.21.15.78:45620) 2023-10-14T11:58:43.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.787+0000 7fb560f95700 1 -- 172.21.15.78:0/4181258991 learned_addr learned my addr 172.21.15.78:0/4181258991 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:43.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.787+0000 7fb55bfff700 1 -- 172.21.15.78:0/4181258991 >> v1:172.21.15.195:6789/0 conn(0x7fb55c14e520 legacy=0x7fb55c0ab960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:43.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.787+0000 7fb55bfff700 1 -- 172.21.15.78:0/4181258991 >> v1:172.21.15.78:6789/0 conn(0x7fb55c0c69e0 legacy=0x7fb55c06e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:43.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.787+0000 7fb55bfff700 1 -- 172.21.15.78:0/4181258991 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb55c070c50 con 0x7fb55c0c11a0 2023-10-14T11:58:43.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.787+0000 7fb55b7fe700 1 -- 172.21.15.78:0/4181258991 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1868501326 0 0) 0x7fb55c1542e0 con 0x7fb55c0c69e0 2023-10-14T11:58:43.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.787+0000 7fb55bfff700 1 --2- 172.21.15.78:0/4181258991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb55c0c11a0 0x7fb55c14dfe0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb548009840 tx=0x7fb548009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f67ccd569d2a6d79 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:43.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.787+0000 7fb55b7fe700 1 -- 172.21.15.78:0/4181258991 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb548004230 con 0x7fb55c0c11a0 2023-10-14T11:58:43.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.787+0000 7fb55b7fe700 1 -- 172.21.15.78:0/4181258991 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb548004390 con 0x7fb55c0c11a0 2023-10-14T11:58:43.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.787+0000 7fb55b7fe700 1 -- 172.21.15.78:0/4181258991 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5480036a0 con 0x7fb55c0c11a0 2023-10-14T11:58:43.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.788+0000 7fb5631f9700 1 -- 172.21.15.78:0/4181258991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb55c0c11a0 msgr2=0x7fb55c14dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:43.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.788+0000 7fb5631f9700 1 --2- 172.21.15.78:0/4181258991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb55c0c11a0 0x7fb55c14dfe0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb548009840 tx=0x7fb548009b50 comp rx=0 tx=0).stop 2023-10-14T11:58:43.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.788+0000 7fb5631f9700 1 -- 172.21.15.78:0/4181258991 shutdown_connections 2023-10-14T11:58:43.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.788+0000 7fb5631f9700 1 --2- 172.21.15.78:0/4181258991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb55c0c11a0 0x7fb55c14dfe0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:43.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.788+0000 7fb5631f9700 1 -- 172.21.15.78:0/4181258991 >> 172.21.15.78:0/4181258991 conn(0x7fb55c0a8410 msgr2=0x7fb55c0a8820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:43.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.788+0000 7fb5631f9700 1 -- 172.21.15.78:0/4181258991 shutdown_connections 2023-10-14T11:58:43.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.788+0000 7fb5631f9700 1 -- 172.21.15.78:0/4181258991 wait complete. 2023-10-14T11:58:43.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.789+0000 7fb5631f9700 1 Processor -- start 2023-10-14T11:58:43.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.789+0000 7fb5631f9700 1 -- start start 2023-10-14T11:58:43.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.790+0000 7fb5631f9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb55c0c11a0 0x7fb55c0bfd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:43.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.790+0000 7fb5631f9700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb55c0c69e0 0x7fb55c14d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:43.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.790+0000 7fb5631f9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb55c14e520 0x7fb55c14d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:43.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.790+0000 7fb5631f9700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55c099fb0 con 0x7fb55c0c11a0 2023-10-14T11:58:43.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.790+0000 7fb5631f9700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55c1d8ff0 con 0x7fb55c14e520 2023-10-14T11:58:43.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.790+0000 7fb5631f9700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb55c1e7820 con 0x7fb55c0c69e0 2023-10-14T11:58:43.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.791+0000 7fb560f95700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb55c0c11a0 0x7fb55c0bfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:43.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.791+0000 7fb560f95700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb55c0c11a0 0x7fb55c0bfd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45624/0 (socket says 172.21.15.78:45624) 2023-10-14T11:58:43.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.791+0000 7fb560f95700 1 -- 172.21.15.78:0/3606536686 learned_addr learned my addr 172.21.15.78:0/3606536686 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:43.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.791+0000 7fb561796700 1 --2- 172.21.15.78:0/3606536686 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb55c14e520 0x7fb55c14d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:43.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.791+0000 7fb55bfff700 1 --2- 172.21.15.78:0/3606536686 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb55c0c69e0 0x7fb55c14d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:43.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.791+0000 7fb560f95700 1 -- 172.21.15.78:0/3606536686 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb55c0c69e0 msgr2=0x7fb55c14d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:43.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.791+0000 7fb560f95700 1 --2- 172.21.15.78:0/3606536686 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb55c0c69e0 0x7fb55c14d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:43.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.791+0000 7fb560f95700 1 -- 172.21.15.78:0/3606536686 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb55c14e520 msgr2=0x7fb55c14d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:43.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.791+0000 7fb560f95700 1 --2- 172.21.15.78:0/3606536686 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb55c14e520 0x7fb55c14d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:43.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.791+0000 7fb560f95700 1 -- 172.21.15.78:0/3606536686 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5480094a0 con 0x7fb55c0c11a0 2023-10-14T11:58:43.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.791+0000 7fb560f95700 1 --2- 172.21.15.78:0/3606536686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb55c0c11a0 0x7fb55c0bfd70 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb5500011e0 tx=0x7fb55000cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:43.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.791+0000 7fb5597fa700 1 -- 172.21.15.78:0/3606536686 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb550004c80 con 0x7fb55c0c11a0 2023-10-14T11:58:43.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.791+0000 7fb5597fa700 1 -- 172.21.15.78:0/3606536686 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb550004e00 con 0x7fb55c0c11a0 2023-10-14T11:58:43.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.791+0000 7fb5597fa700 1 -- 172.21.15.78:0/3606536686 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb550007310 con 0x7fb55c0c11a0 2023-10-14T11:58:43.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.791+0000 7fb5631f9700 1 -- 172.21.15.78:0/3606536686 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb55c1ebb10 con 0x7fb55c0c11a0 2023-10-14T11:58:43.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.791+0000 7fb5631f9700 1 -- 172.21.15.78:0/3606536686 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb55c1d9520 con 0x7fb55c0c11a0 2023-10-14T11:58:43.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.792+0000 7fb5631f9700 1 -- 172.21.15.78:0/3606536686 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb55c1e7a40 con 0x7fb55c0c11a0 2023-10-14T11:58:43.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.793+0000 7fb5597fa700 1 -- 172.21.15.78:0/3606536686 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb550007470 con 0x7fb55c0c11a0 2023-10-14T11:58:43.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.793+0000 7fb5597fa700 1 --2- 172.21.15.78:0/3606536686 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb54006af00 0x7fb54006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:43.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.793+0000 7fb5597fa700 1 -- 172.21.15.78:0/3606536686 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb550089380 con 0x7fb55c0c11a0 2023-10-14T11:58:43.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.793+0000 7fb55bfff700 1 --2- 172.21.15.78:0/3606536686 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb54006af00 0x7fb54006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:43.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.793+0000 7fb55bfff700 1 --2- 172.21.15.78:0/3606536686 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb54006af00 0x7fb54006d3c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb548000c00 tx=0x7fb54801c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:43.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.797+0000 7fb5597fa700 1 -- 172.21.15.78:0/3606536686 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb55c1e7a40 con 0x7fb55c0c11a0 2023-10-14T11:58:43.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.984+0000 7fb5631f9700 1 -- 172.21.15.78:0/3606536686 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb55c074b80 con 0x7fb55c0c11a0 2023-10-14T11:58:43.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.984+0000 7fb5597fa700 1 -- 172.21.15.78:0/3606536686 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb5500544c0 con 0x7fb55c0c11a0 2023-10-14T11:58:43.987 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-10-14T11:58:43.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.986+0000 7fb5631f9700 1 -- 172.21.15.78:0/3606536686 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb54006af00 msgr2=0x7fb54006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:43.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.986+0000 7fb5631f9700 1 --2- 172.21.15.78:0/3606536686 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb54006af00 0x7fb54006d3c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb548000c00 tx=0x7fb54801c040 comp rx=0 tx=0).stop 2023-10-14T11:58:43.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.986+0000 7fb5631f9700 1 -- 172.21.15.78:0/3606536686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb55c0c11a0 msgr2=0x7fb55c0bfd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:43.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.986+0000 7fb5631f9700 1 --2- 172.21.15.78:0/3606536686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb55c0c11a0 0x7fb55c0bfd70 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb5500011e0 tx=0x7fb55000cae0 comp rx=0 tx=0).stop 2023-10-14T11:58:43.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.987+0000 7fb5631f9700 1 -- 172.21.15.78:0/3606536686 shutdown_connections 2023-10-14T11:58:43.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.987+0000 7fb5631f9700 1 --2- 172.21.15.78:0/3606536686 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb54006af00 0x7fb54006d3c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:43.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.987+0000 7fb5631f9700 1 --2- 172.21.15.78:0/3606536686 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb55c14e520 0x7fb55c14d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:43.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.987+0000 7fb5631f9700 1 --2- 172.21.15.78:0/3606536686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb55c0c11a0 0x7fb55c0bfd70 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:43.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.987+0000 7fb5631f9700 1 --2- 172.21.15.78:0/3606536686 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb55c0c69e0 0x7fb55c14d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:43.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.987+0000 7fb5631f9700 1 -- 172.21.15.78:0/3606536686 >> 172.21.15.78:0/3606536686 conn(0x7fb55c0a8410 msgr2=0x7fb55c0a87f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:43.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.987+0000 7fb5631f9700 1 -- 172.21.15.78:0/3606536686 shutdown_connections 2023-10-14T11:58:43.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:43.987+0000 7fb5631f9700 1 -- 172.21.15.78:0/3606536686 wait complete. 2023-10-14T11:58:44.000 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T11:58:44.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.121+0000 7fb6a9986700 1 Processor -- start 2023-10-14T11:58:44.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.121+0000 7fb6a9986700 1 -- start start 2023-10-14T11:58:44.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.122+0000 7fb6a9986700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb6a40c6cd0 0x7fb6a40c70b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:44.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.122+0000 7fb6a9986700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb6a40bd950 0x7fb6a41588e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:44.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.122+0000 7fb6a9986700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6a40c4c40 con 0x7fb6a414ebd0 2023-10-14T11:58:44.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.122+0000 7fb6a9986700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6a4158e20 con 0x7fb6a40bd950 2023-10-14T11:58:44.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.122+0000 7fb6a9986700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6a40c54e0 con 0x7fb6a40c6cd0 2023-10-14T11:58:44.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.122+0000 7fb6a37fe700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb6a40bd950 0x7fb6a41588e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:44.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.122+0000 7fb6a27fc700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb6a414ebd0 0x7fb6a41581d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50510/0 (socket says 172.21.15.78:50510) 2023-10-14T11:58:44.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.122+0000 7fb6a37fe700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb6a40bd950 0x7fb6a41588e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45636/0 (socket says 172.21.15.78:45636) 2023-10-14T11:58:44.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.122+0000 7fb6a27fc700 1 -- 172.21.15.78:0/3146844114 learned_addr learned my addr 172.21.15.78:0/3146844114 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:44.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.122+0000 7fb6a2ffd700 1 --2- 172.21.15.78:0/3146844114 >> v2:172.21.15.195:3300/0 conn(0x7fb6a40c6cd0 0x7fb6a40c70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:44.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.122+0000 7fb6a37fe700 1 -- 172.21.15.78:0/3146844114 >> v2:172.21.15.195:3300/0 conn(0x7fb6a40c6cd0 msgr2=0x7fb6a40c70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:44.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.122+0000 7fb6a37fe700 1 --2- 172.21.15.78:0/3146844114 >> v2:172.21.15.195:3300/0 conn(0x7fb6a40c6cd0 0x7fb6a40c70b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:44.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.122+0000 7fb6a37fe700 1 -- 172.21.15.78:0/3146844114 >> v1:172.21.15.78:6789/0 conn(0x7fb6a414ebd0 legacy=0x7fb6a41581d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:44.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.122+0000 7fb6a37fe700 1 -- 172.21.15.78:0/3146844114 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a41556d0 con 0x7fb6a40bd950 2023-10-14T11:58:44.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.122+0000 7fb6a1ffb700 1 -- 172.21.15.78:0/3146844114 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2262256156 0 0) 0x7fb6a40c4c40 con 0x7fb6a414ebd0 2023-10-14T11:58:44.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.122+0000 7fb6a37fe700 1 --2- 172.21.15.78:0/3146844114 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6a40bd950 0x7fb6a41588e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb6a406c9e0 tx=0x7fb69c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd9a12c62628d726 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:44.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.123+0000 7fb6a1ffb700 1 -- 172.21.15.78:0/3146844114 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69c00e070 con 0x7fb6a40bd950 2023-10-14T11:58:44.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.123+0000 7fb6a1ffb700 1 -- 172.21.15.78:0/3146844114 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb69c00bbd0 con 0x7fb6a40bd950 2023-10-14T11:58:44.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.123+0000 7fb6a1ffb700 1 -- 172.21.15.78:0/3146844114 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69c012430 con 0x7fb6a40bd950 2023-10-14T11:58:44.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.123+0000 7fb6a9986700 1 -- 172.21.15.78:0/3146844114 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6a40bd950 msgr2=0x7fb6a41588e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:44.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.123+0000 7fb6a9986700 1 --2- 172.21.15.78:0/3146844114 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6a40bd950 0x7fb6a41588e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb6a406c9e0 tx=0x7fb69c00b330 comp rx=0 tx=0).stop 2023-10-14T11:58:44.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.123+0000 7fb6a9986700 1 -- 172.21.15.78:0/3146844114 shutdown_connections 2023-10-14T11:58:44.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.123+0000 7fb6a9986700 1 --2- 172.21.15.78:0/3146844114 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6a40bd950 0x7fb6a41588e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:44.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.123+0000 7fb6a9986700 1 --2- 172.21.15.78:0/3146844114 >> v2:172.21.15.195:3300/0 conn(0x7fb6a40c6cd0 0x7fb6a40c70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:44.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.123+0000 7fb6a9986700 1 -- 172.21.15.78:0/3146844114 >> 172.21.15.78:0/3146844114 conn(0x7fb6a40abdf0 msgr2=0x7fb6a40ac1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:44.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.123+0000 7fb6a9986700 1 -- 172.21.15.78:0/3146844114 shutdown_connections 2023-10-14T11:58:44.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.123+0000 7fb6a9986700 1 -- 172.21.15.78:0/3146844114 wait complete. 2023-10-14T11:58:44.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.124+0000 7fb6a9986700 1 Processor -- start 2023-10-14T11:58:44.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.124+0000 7fb6a9986700 1 -- start start 2023-10-14T11:58:44.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.125+0000 7fb6a9986700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6a40bd950 0x7fb6a40c1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:44.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.125+0000 7fb6a9986700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6a40c6cd0 0x7fb6a41eaa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:44.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.125+0000 7fb6a9986700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb6a4153fd0 0x7fb6a41ede20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:44.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.125+0000 7fb6a9986700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a41eafb0 con 0x7fb6a40bd950 2023-10-14T11:58:44.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.125+0000 7fb6a9986700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a41575e0 con 0x7fb6a40c6cd0 2023-10-14T11:58:44.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.125+0000 7fb6a9986700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a41578c0 con 0x7fb6a4153fd0 2023-10-14T11:58:44.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.125+0000 7fb6a2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6a40bd950 0x7fb6a40c1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:44.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.125+0000 7fb6a2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6a40bd950 0x7fb6a40c1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45638/0 (socket says 172.21.15.78:45638) 2023-10-14T11:58:44.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.125+0000 7fb6a2ffd700 1 -- 172.21.15.78:0/2531565784 learned_addr learned my addr 172.21.15.78:0/2531565784 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:44.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.125+0000 7fb6a2ffd700 1 -- 172.21.15.78:0/2531565784 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb6a4153fd0 msgr2=0x7fb6a41ede20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:44.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.125+0000 7fb6a2ffd700 1 --2- 172.21.15.78:0/2531565784 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb6a4153fd0 0x7fb6a41ede20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:44.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.125+0000 7fb6a2ffd700 1 -- 172.21.15.78:0/2531565784 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6a40c6cd0 msgr2=0x7fb6a41eaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:44.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.125+0000 7fb6a2ffd700 1 --2- 172.21.15.78:0/2531565784 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6a40c6cd0 0x7fb6a41eaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:44.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.126+0000 7fb6a2ffd700 1 -- 172.21.15.78:0/2531565784 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb69c009a70 con 0x7fb6a40bd950 2023-10-14T11:58:44.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.126+0000 7fb6a2ffd700 1 --2- 172.21.15.78:0/2531565784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6a40bd950 0x7fb6a40c1a10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb69800c7f0 tx=0x7fb69800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:44.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.126+0000 7fb68bfff700 1 -- 172.21.15.78:0/2531565784 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb698011800 con 0x7fb6a40bd950 2023-10-14T11:58:44.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.126+0000 7fb68bfff700 1 -- 172.21.15.78:0/2531565784 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb698011960 con 0x7fb6a40bd950 2023-10-14T11:58:44.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.126+0000 7fb6a9986700 1 -- 172.21.15.78:0/2531565784 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6a4157ba0 con 0x7fb6a40bd950 2023-10-14T11:58:44.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.126+0000 7fb68bfff700 1 -- 172.21.15.78:0/2531565784 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb698011cf0 con 0x7fb6a40bd950 2023-10-14T11:58:44.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.126+0000 7fb6a9986700 1 -- 172.21.15.78:0/2531565784 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6a41ed900 con 0x7fb6a40bd950 2023-10-14T11:58:44.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.127+0000 7fb6a9986700 1 -- 172.21.15.78:0/2531565784 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6a41edb20 con 0x7fb6a40bd950 2023-10-14T11:58:44.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.127+0000 7fb68bfff700 1 -- 172.21.15.78:0/2531565784 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb698018360 con 0x7fb6a40bd950 2023-10-14T11:58:44.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.128+0000 7fb68bfff700 1 --2- 172.21.15.78:0/2531565784 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb69006af00 0x7fb69006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:44.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.128+0000 7fb68bfff700 1 -- 172.21.15.78:0/2531565784 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb698089560 con 0x7fb6a40bd950 2023-10-14T11:58:44.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.128+0000 7fb6a27fc700 1 --2- 172.21.15.78:0/2531565784 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb69006af00 0x7fb69006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:44.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.128+0000 7fb6a27fc700 1 --2- 172.21.15.78:0/2531565784 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb69006af00 0x7fb69006d3c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb6a40c19c0 tx=0x7fb6940079f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:44.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.132+0000 7fb68bfff700 1 -- 172.21.15.78:0/2531565784 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb698054620 con 0x7fb6a40bd950 2023-10-14T11:58:44.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.319+0000 7fb6a9986700 1 -- 172.21.15.78:0/2531565784 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb6a40683e0 con 0x7fb6a40bd950 2023-10-14T11:58:44.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.320+0000 7fb68bfff700 1 -- 172.21.15.78:0/2531565784 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb69805e020 con 0x7fb6a40bd950 2023-10-14T11:58:44.322 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-10-14T11:58:44.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.322+0000 7fb6a9986700 1 -- 172.21.15.78:0/2531565784 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb69006af00 msgr2=0x7fb69006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:44.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.322+0000 7fb6a9986700 1 --2- 172.21.15.78:0/2531565784 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb69006af00 0x7fb69006d3c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb6a40c19c0 tx=0x7fb6940079f0 comp rx=0 tx=0).stop 2023-10-14T11:58:44.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.322+0000 7fb6a9986700 1 -- 172.21.15.78:0/2531565784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6a40bd950 msgr2=0x7fb6a40c1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:44.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.322+0000 7fb6a9986700 1 --2- 172.21.15.78:0/2531565784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6a40bd950 0x7fb6a40c1a10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb69800c7f0 tx=0x7fb69800cbb0 comp rx=0 tx=0).stop 2023-10-14T11:58:44.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.322+0000 7fb6a9986700 1 -- 172.21.15.78:0/2531565784 shutdown_connections 2023-10-14T11:58:44.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.322+0000 7fb6a9986700 1 --2- 172.21.15.78:0/2531565784 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb6a4153fd0 0x7fb6a41ede20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:44.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.322+0000 7fb6a9986700 1 --2- 172.21.15.78:0/2531565784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6a40bd950 0x7fb6a40c1a10 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:44.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.322+0000 7fb6a9986700 1 --2- 172.21.15.78:0/2531565784 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb69006af00 0x7fb69006d3c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:44.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.322+0000 7fb6a9986700 1 --2- 172.21.15.78:0/2531565784 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6a40c6cd0 0x7fb6a41eaa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:44.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.322+0000 7fb6a9986700 1 -- 172.21.15.78:0/2531565784 >> 172.21.15.78:0/2531565784 conn(0x7fb6a40abdf0 msgr2=0x7fb6a406b610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:44.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.322+0000 7fb6a9986700 1 -- 172.21.15.78:0/2531565784 shutdown_connections 2023-10-14T11:58:44.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.322+0000 7fb6a9986700 1 -- 172.21.15.78:0/2531565784 wait complete. 2023-10-14T11:58:44.336 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T11:58:44.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.457+0000 7f43dc875700 1 Processor -- start 2023-10-14T11:58:44.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.457+0000 7f43dc875700 1 -- start start 2023-10-14T11:58:44.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.458+0000 7f43dc875700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f43d806dc80 0x7f43d806e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:44.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.458+0000 7f43dc875700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f43d8071480 0x7f43d80a9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:44.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.458+0000 7f43dc875700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f43d80aa150 0x7f43d8155050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:44.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.458+0000 7f43dc875700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43d80c1c60 con 0x7f43d8071480 2023-10-14T11:58:44.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.458+0000 7f43dc875700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43d80c1800 con 0x7f43d80aa150 2023-10-14T11:58:44.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.458+0000 7f43dc875700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43d80c3b90 con 0x7f43d806dc80 2023-10-14T11:58:44.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.458+0000 7f43d5d9b700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f43d8071480 0x7f43d80a9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:44.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.458+0000 7f43d5d9b700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f43d8071480 0x7f43d80a9c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45646/0 (socket says 172.21.15.78:45646) 2023-10-14T11:58:44.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.458+0000 7f43d5d9b700 1 -- 172.21.15.78:0/710815865 learned_addr learned my addr 172.21.15.78:0/710815865 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:44.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.459+0000 7f43d6d9d700 1 --2- 172.21.15.78:0/710815865 >> v2:172.21.15.195:3300/0 conn(0x7f43d80aa150 0x7f43d8155050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:44.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.459+0000 7f43d659c700 1 --2- 172.21.15.78:0/710815865 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f43d806dc80 0x7f43d806e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:44.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.459+0000 7f43d5d9b700 1 -- 172.21.15.78:0/710815865 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f43d806dc80 msgr2=0x7f43d806e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:44.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.459+0000 7f43d5d9b700 1 --2- 172.21.15.78:0/710815865 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f43d806dc80 0x7f43d806e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:44.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.459+0000 7f43d5d9b700 1 -- 172.21.15.78:0/710815865 >> v2:172.21.15.195:3300/0 conn(0x7f43d80aa150 msgr2=0x7f43d8155050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:44.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.459+0000 7f43d5d9b700 1 --2- 172.21.15.78:0/710815865 >> v2:172.21.15.195:3300/0 conn(0x7f43d80aa150 0x7f43d8155050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:44.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.459+0000 7f43d5d9b700 1 -- 172.21.15.78:0/710815865 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d80c1240 con 0x7f43d8071480 2023-10-14T11:58:44.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.459+0000 7f43d6d9d700 1 --2- 172.21.15.78:0/710815865 >> v2:172.21.15.195:3300/0 conn(0x7f43d80aa150 0x7f43d8155050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:44.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.459+0000 7f43d5d9b700 1 --2- 172.21.15.78:0/710815865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43d8071480 0x7f43d80a9c10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f43d8073a20 tx=0x7f43c8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=405b2e89c32d18db server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:44.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.459+0000 7f43d559a700 1 -- 172.21.15.78:0/710815865 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c800b860 con 0x7f43d8071480 2023-10-14T11:58:44.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.459+0000 7f43d559a700 1 -- 172.21.15.78:0/710815865 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43c800b9c0 con 0x7f43d8071480 2023-10-14T11:58:44.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.459+0000 7f43d559a700 1 -- 172.21.15.78:0/710815865 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c800bd50 con 0x7f43d8071480 2023-10-14T11:58:44.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.460+0000 7f43dc875700 1 -- 172.21.15.78:0/710815865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43d8071480 msgr2=0x7f43d80a9c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:44.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.460+0000 7f43dc875700 1 --2- 172.21.15.78:0/710815865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43d8071480 0x7f43d80a9c10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f43d8073a20 tx=0x7f43c8009c00 comp rx=0 tx=0).stop 2023-10-14T11:58:44.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.460+0000 7f43dc875700 1 -- 172.21.15.78:0/710815865 shutdown_connections 2023-10-14T11:58:44.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.460+0000 7f43dc875700 1 --2- 172.21.15.78:0/710815865 >> v2:172.21.15.195:3300/0 conn(0x7f43d80aa150 0x7f43d8155050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:44.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.460+0000 7f43dc875700 1 --2- 172.21.15.78:0/710815865 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f43d806dc80 0x7f43d806e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:44.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.460+0000 7f43dc875700 1 --2- 172.21.15.78:0/710815865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43d8071480 0x7f43d80a9c10 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:44.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.460+0000 7f43dc875700 1 -- 172.21.15.78:0/710815865 >> 172.21.15.78:0/710815865 conn(0x7f43d80a8410 msgr2=0x7f43d80a8820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:44.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.460+0000 7f43dc875700 1 -- 172.21.15.78:0/710815865 shutdown_connections 2023-10-14T11:58:44.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.460+0000 7f43dc875700 1 -- 172.21.15.78:0/710815865 wait complete. 2023-10-14T11:58:44.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.461+0000 7f43dc875700 1 Processor -- start 2023-10-14T11:58:44.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.461+0000 7f43dc875700 1 -- start start 2023-10-14T11:58:44.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.462+0000 7f43dc875700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f43d806dc80 0x7f43d81d4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:44.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.462+0000 7f43dc875700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f43d8071480 0x7f43d81e5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:44.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.462+0000 7f43dc875700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43d80aa150 0x7f43d81d3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:44.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.462+0000 7f43dc875700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d81e9bf0 con 0x7f43d80aa150 2023-10-14T11:58:44.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.462+0000 7f43dc875700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d81e5bc0 con 0x7f43d8071480 2023-10-14T11:58:44.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.462+0000 7f43dc875700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43d80baff0 con 0x7f43d806dc80 2023-10-14T11:58:44.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.462+0000 7f43d6d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43d80aa150 0x7f43d81d3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:44.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.462+0000 7f43d6d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43d80aa150 0x7f43d81d3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45656/0 (socket says 172.21.15.78:45656) 2023-10-14T11:58:44.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.462+0000 7f43d6d9d700 1 -- 172.21.15.78:0/2493302203 learned_addr learned my addr 172.21.15.78:0/2493302203 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:44.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.462+0000 7f43d659c700 1 --2- 172.21.15.78:0/2493302203 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f43d806dc80 0x7f43d81d4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:44.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.462+0000 7f43d5d9b700 1 --2- 172.21.15.78:0/2493302203 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f43d8071480 0x7f43d81e5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:44.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.462+0000 7f43d6d9d700 1 -- 172.21.15.78:0/2493302203 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f43d806dc80 msgr2=0x7f43d81d4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:44.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.462+0000 7f43d6d9d700 1 --2- 172.21.15.78:0/2493302203 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f43d806dc80 0x7f43d81d4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:44.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.462+0000 7f43d6d9d700 1 -- 172.21.15.78:0/2493302203 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f43d8071480 msgr2=0x7f43d81e5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:44.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.462+0000 7f43d6d9d700 1 --2- 172.21.15.78:0/2493302203 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f43d8071480 0x7f43d81e5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:44.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.462+0000 7f43d6d9d700 1 -- 172.21.15.78:0/2493302203 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c8009550 con 0x7f43d80aa150 2023-10-14T11:58:44.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.462+0000 7f43d659c700 1 --2- 172.21.15.78:0/2493302203 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f43d806dc80 0x7f43d81d4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:44.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.462+0000 7f43d5d9b700 1 --2- 172.21.15.78:0/2493302203 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f43d8071480 0x7f43d81e5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:44.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.462+0000 7f43d6d9d700 1 --2- 172.21.15.78:0/2493302203 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43d80aa150 0x7f43d81d3d00 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f43d000cef0 tx=0x7f43d000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:44.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.462+0000 7f43c6ffd700 1 -- 172.21.15.78:0/2493302203 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d000ae50 con 0x7f43d80aa150 2023-10-14T11:58:44.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.462+0000 7f43c6ffd700 1 -- 172.21.15.78:0/2493302203 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d0015440 con 0x7f43d80aa150 2023-10-14T11:58:44.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.462+0000 7f43c6ffd700 1 -- 172.21.15.78:0/2493302203 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d00157f0 con 0x7f43d80aa150 2023-10-14T11:58:44.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.462+0000 7f43dc875700 1 -- 172.21.15.78:0/2493302203 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43d80bbf60 con 0x7f43d80aa150 2023-10-14T11:58:44.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.462+0000 7f43dc875700 1 -- 172.21.15.78:0/2493302203 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43d80ba480 con 0x7f43d80aa150 2023-10-14T11:58:44.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.463+0000 7f43dc875700 1 -- 172.21.15.78:0/2493302203 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43d80ba670 con 0x7f43d80aa150 2023-10-14T11:58:44.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.464+0000 7f43c6ffd700 1 -- 172.21.15.78:0/2493302203 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f43d000d430 con 0x7f43d80aa150 2023-10-14T11:58:44.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.464+0000 7f43c6ffd700 1 --2- 172.21.15.78:0/2493302203 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f43b806af00 0x7f43b806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:44.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.464+0000 7f43d659c700 1 --2- 172.21.15.78:0/2493302203 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f43b806af00 0x7f43b806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:44.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.464+0000 7f43c6ffd700 1 -- 172.21.15.78:0/2493302203 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f43d0089be0 con 0x7f43d80aa150 2023-10-14T11:58:44.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.465+0000 7f43d659c700 1 --2- 172.21.15.78:0/2493302203 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f43b806af00 0x7f43b806d3c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f43cc00a990 tx=0x7f43cc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:44.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.469+0000 7f43c6ffd700 1 -- 172.21.15.78:0/2493302203 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f43d0054c80 con 0x7f43d80aa150 2023-10-14T11:58:44.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.656+0000 7f43dc875700 1 -- 172.21.15.78:0/2493302203 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f43d81e5d00 con 0x7f43d80aa150 2023-10-14T11:58:44.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.656+0000 7f43c6ffd700 1 -- 172.21.15.78:0/2493302203 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f43d001a020 con 0x7f43d80aa150 2023-10-14T11:58:44.659 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2023-10-14T11:58:44.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.658+0000 7f43dc875700 1 -- 172.21.15.78:0/2493302203 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f43b806af00 msgr2=0x7f43b806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:44.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.658+0000 7f43dc875700 1 --2- 172.21.15.78:0/2493302203 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f43b806af00 0x7f43b806d3c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f43cc00a990 tx=0x7f43cc008040 comp rx=0 tx=0).stop 2023-10-14T11:58:44.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.658+0000 7f43dc875700 1 -- 172.21.15.78:0/2493302203 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43d80aa150 msgr2=0x7f43d81d3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:44.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.658+0000 7f43dc875700 1 --2- 172.21.15.78:0/2493302203 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43d80aa150 0x7f43d81d3d00 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f43d000cef0 tx=0x7f43d000a340 comp rx=0 tx=0).stop 2023-10-14T11:58:44.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.658+0000 7f43dc875700 1 -- 172.21.15.78:0/2493302203 shutdown_connections 2023-10-14T11:58:44.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.658+0000 7f43dc875700 1 --2- 172.21.15.78:0/2493302203 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43d80aa150 0x7f43d81d3d00 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:44.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.658+0000 7f43dc875700 1 --2- 172.21.15.78:0/2493302203 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f43d806dc80 0x7f43d81d4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:44.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.658+0000 7f43dc875700 1 --2- 172.21.15.78:0/2493302203 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f43b806af00 0x7f43b806d3c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:44.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.658+0000 7f43dc875700 1 --2- 172.21.15.78:0/2493302203 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f43d8071480 0x7f43d81e5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:44.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.658+0000 7f43dc875700 1 -- 172.21.15.78:0/2493302203 >> 172.21.15.78:0/2493302203 conn(0x7f43d80a8410 msgr2=0x7f43d81db4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:44.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.659+0000 7f43dc875700 1 -- 172.21.15.78:0/2493302203 shutdown_connections 2023-10-14T11:58:44.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.659+0000 7f43dc875700 1 -- 172.21.15.78:0/2493302203 wait complete. 2023-10-14T11:58:44.672 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-14T11:58:44.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.792+0000 7f34e13b3700 1 Processor -- start 2023-10-14T11:58:44.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.792+0000 7f34e13b3700 1 -- start start 2023-10-14T11:58:44.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.792+0000 7f34e13b3700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f34dc1533d0 0x7f34dc0c8680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:44.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.792+0000 7f34e13b3700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f34dc0a70c0 0x7f34dc155070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:44.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.792+0000 7f34e13b3700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34dc1537b0 con 0x7f34dc0c8bc0 2023-10-14T11:58:44.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.792+0000 7f34e13b3700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34dc0c6fb0 con 0x7f34dc1533d0 2023-10-14T11:58:44.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.792+0000 7f34e13b3700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34dc0c7f30 con 0x7f34dc0a70c0 2023-10-14T11:58:44.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.793+0000 7f34daffd700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f34dc1533d0 0x7f34dc0c8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:44.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.793+0000 7f34daffd700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f34dc1533d0 0x7f34dc0c8680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45660/0 (socket says 172.21.15.78:45660) 2023-10-14T11:58:44.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.793+0000 7f34da7fc700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f34dc0c8bc0 0x7f34dc154960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50536/0 (socket says 172.21.15.78:50536) 2023-10-14T11:58:44.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.793+0000 7f34daffd700 1 -- 172.21.15.78:0/2023591529 learned_addr learned my addr 172.21.15.78:0/2023591529 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:44.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.793+0000 7f34db7fe700 1 --2- 172.21.15.78:0/2023591529 >> v2:172.21.15.195:3300/0 conn(0x7f34dc0a70c0 0x7f34dc155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:44.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.793+0000 7f34daffd700 1 -- 172.21.15.78:0/2023591529 >> v2:172.21.15.195:3300/0 conn(0x7f34dc0a70c0 msgr2=0x7f34dc155070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:44.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.793+0000 7f34daffd700 1 --2- 172.21.15.78:0/2023591529 >> v2:172.21.15.195:3300/0 conn(0x7f34dc0a70c0 0x7f34dc155070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:44.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.793+0000 7f34daffd700 1 -- 172.21.15.78:0/2023591529 >> v1:172.21.15.78:6789/0 conn(0x7f34dc0c8bc0 legacy=0x7f34dc154960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:44.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.793+0000 7f34daffd700 1 -- 172.21.15.78:0/2023591529 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34dc0c7c50 con 0x7f34dc1533d0 2023-10-14T11:58:44.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.793+0000 7f34d9ffb700 1 -- 172.21.15.78:0/2023591529 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1974669119 0 0) 0x7f34dc1537b0 con 0x7f34dc0c8bc0 2023-10-14T11:58:44.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.793+0000 7f34db7fe700 1 --2- 172.21.15.78:0/2023591529 >> v2:172.21.15.195:3300/0 conn(0x7f34dc0a70c0 0x7f34dc155070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:44.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.793+0000 7f34daffd700 1 --2- 172.21.15.78:0/2023591529 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34dc1533d0 0x7f34dc0c8680 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f34dc156dc0 tx=0x7f34cc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cddcb3f071e378ac server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:44.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.793+0000 7f34d9ffb700 1 -- 172.21.15.78:0/2023591529 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34cc00b870 con 0x7f34dc1533d0 2023-10-14T11:58:44.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.793+0000 7f34d9ffb700 1 -- 172.21.15.78:0/2023591529 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34cc00b9d0 con 0x7f34dc1533d0 2023-10-14T11:58:44.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.793+0000 7f34d9ffb700 1 -- 172.21.15.78:0/2023591529 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34cc00bd60 con 0x7f34dc1533d0 2023-10-14T11:58:44.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.794+0000 7f34e13b3700 1 -- 172.21.15.78:0/2023591529 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34dc1533d0 msgr2=0x7f34dc0c8680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:44.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.794+0000 7f34e13b3700 1 --2- 172.21.15.78:0/2023591529 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34dc1533d0 0x7f34dc0c8680 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f34dc156dc0 tx=0x7f34cc009bd0 comp rx=0 tx=0).stop 2023-10-14T11:58:44.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.794+0000 7f34e13b3700 1 -- 172.21.15.78:0/2023591529 shutdown_connections 2023-10-14T11:58:44.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.794+0000 7f34e13b3700 1 --2- 172.21.15.78:0/2023591529 >> v2:172.21.15.195:3300/0 conn(0x7f34dc0a70c0 0x7f34dc155070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:44.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.794+0000 7f34e13b3700 1 --2- 172.21.15.78:0/2023591529 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34dc1533d0 0x7f34dc0c8680 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:44.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.794+0000 7f34e13b3700 1 -- 172.21.15.78:0/2023591529 >> 172.21.15.78:0/2023591529 conn(0x7f34dc0737d0 msgr2=0x7f34dc073be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:44.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.794+0000 7f34e13b3700 1 -- 172.21.15.78:0/2023591529 shutdown_connections 2023-10-14T11:58:44.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.794+0000 7f34e13b3700 1 -- 172.21.15.78:0/2023591529 wait complete. 2023-10-14T11:58:44.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.795+0000 7f34e13b3700 1 Processor -- start 2023-10-14T11:58:44.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.795+0000 7f34e13b3700 1 -- start start 2023-10-14T11:58:44.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.796+0000 7f34e13b3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34dc0a70c0 0x7f34dc14b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:44.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.796+0000 7f34e13b3700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f34dc0c8bc0 0x7f34dc0c1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:44.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.796+0000 7f34e13b3700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f34dc1533d0 0x7f34dc0c1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:44.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.796+0000 7f34e13b3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34dc0c7a70 con 0x7f34dc0a70c0 2023-10-14T11:58:44.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.796+0000 7f34e13b3700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34dc0c1cd0 con 0x7f34dc1533d0 2023-10-14T11:58:44.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.796+0000 7f34e13b3700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34dc0aad40 con 0x7f34dc0c8bc0 2023-10-14T11:58:44.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.796+0000 7f34daffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34dc0a70c0 0x7f34dc14b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:44.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.796+0000 7f34daffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34dc0a70c0 0x7f34dc14b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45666/0 (socket says 172.21.15.78:45666) 2023-10-14T11:58:44.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.796+0000 7f34daffd700 1 -- 172.21.15.78:0/3613593099 learned_addr learned my addr 172.21.15.78:0/3613593099 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:44.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.796+0000 7f34daffd700 1 -- 172.21.15.78:0/3613593099 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f34dc0c8bc0 msgr2=0x7f34dc0c1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:44.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.796+0000 7f34daffd700 1 --2- 172.21.15.78:0/3613593099 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f34dc0c8bc0 0x7f34dc0c1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:44.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.796+0000 7f34daffd700 1 -- 172.21.15.78:0/3613593099 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f34dc1533d0 msgr2=0x7f34dc0c1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:44.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.796+0000 7f34db7fe700 1 --2- 172.21.15.78:0/3613593099 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f34dc1533d0 0x7f34dc0c1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:44.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.796+0000 7f34daffd700 1 --2- 172.21.15.78:0/3613593099 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f34dc1533d0 0x7f34dc0c1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:44.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.796+0000 7f34daffd700 1 -- 172.21.15.78:0/3613593099 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34cc009550 con 0x7f34dc0a70c0 2023-10-14T11:58:44.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.796+0000 7f34daffd700 1 --2- 172.21.15.78:0/3613593099 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34dc0a70c0 0x7f34dc14b300 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f34dc0c5ce0 tx=0x7f34cc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:44.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.796+0000 7f34c37fe700 1 -- 172.21.15.78:0/3613593099 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34cc0117a0 con 0x7f34dc0a70c0 2023-10-14T11:58:44.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.796+0000 7f34c37fe700 1 -- 172.21.15.78:0/3613593099 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34cc01f070 con 0x7f34dc0a70c0 2023-10-14T11:58:44.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.796+0000 7f34e13b3700 1 -- 172.21.15.78:0/3613593099 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34dc1e8520 con 0x7f34dc0a70c0 2023-10-14T11:58:44.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.796+0000 7f34c37fe700 1 -- 172.21.15.78:0/3613593099 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34cc011b00 con 0x7f34dc0a70c0 2023-10-14T11:58:44.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.796+0000 7f34e13b3700 1 -- 172.21.15.78:0/3613593099 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34dc14e4e0 con 0x7f34dc0a70c0 2023-10-14T11:58:44.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.797+0000 7f34e13b3700 1 -- 172.21.15.78:0/3613593099 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34dc14e340 con 0x7f34dc0a70c0 2023-10-14T11:58:44.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.798+0000 7f34c37fe700 1 -- 172.21.15.78:0/3613593099 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f34cc021030 con 0x7f34dc0a70c0 2023-10-14T11:58:44.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.798+0000 7f34c37fe700 1 --2- 172.21.15.78:0/3613593099 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f34c806af00 0x7f34c806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:44.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.798+0000 7f34c37fe700 1 -- 172.21.15.78:0/3613593099 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f34cc08aa80 con 0x7f34dc0a70c0 2023-10-14T11:58:44.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.798+0000 7f34da7fc700 1 --2- 172.21.15.78:0/3613593099 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f34c806af00 0x7f34c806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:44.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.799+0000 7f34da7fc700 1 --2- 172.21.15.78:0/3613593099 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f34c806af00 0x7f34c806d3c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f34d00011e0 tx=0x7f34d00079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:44.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:44.802+0000 7f34c37fe700 1 -- 172.21.15.78:0/3613593099 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f34dc14e340 con 0x7f34dc0a70c0 2023-10-14T11:58:45.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.008+0000 7f34e13b3700 1 -- 172.21.15.78:0/3613593099 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f34dc14e340 con 0x7f34dc0a70c0 2023-10-14T11:58:45.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.008+0000 7f34c37fe700 1 -- 172.21.15.78:0/3613593099 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7f34dc14e340 con 0x7f34dc0a70c0 2023-10-14T11:58:45.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.010+0000 7f34e13b3700 1 -- 172.21.15.78:0/3613593099 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f34c806af00 msgr2=0x7f34c806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:45.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.010+0000 7f34e13b3700 1 --2- 172.21.15.78:0/3613593099 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f34c806af00 0x7f34c806d3c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f34d00011e0 tx=0x7f34d00079b0 comp rx=0 tx=0).stop 2023-10-14T11:58:45.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.010+0000 7f34e13b3700 1 -- 172.21.15.78:0/3613593099 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34dc0a70c0 msgr2=0x7f34dc14b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:45.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.010+0000 7f34e13b3700 1 --2- 172.21.15.78:0/3613593099 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34dc0a70c0 0x7f34dc14b300 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f34dc0c5ce0 tx=0x7f34cc004750 comp rx=0 tx=0).stop 2023-10-14T11:58:45.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.010+0000 7f34e13b3700 1 -- 172.21.15.78:0/3613593099 shutdown_connections 2023-10-14T11:58:45.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.010+0000 7f34e13b3700 1 --2- 172.21.15.78:0/3613593099 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f34c806af00 0x7f34c806d3c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:45.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.010+0000 7f34e13b3700 1 --2- 172.21.15.78:0/3613593099 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f34dc1533d0 0x7f34dc0c1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:45.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.010+0000 7f34e13b3700 1 --2- 172.21.15.78:0/3613593099 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34dc0a70c0 0x7f34dc14b300 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:45.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.010+0000 7f34e13b3700 1 --2- 172.21.15.78:0/3613593099 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f34dc0c8bc0 0x7f34dc0c1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:45.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.010+0000 7f34e13b3700 1 -- 172.21.15.78:0/3613593099 >> 172.21.15.78:0/3613593099 conn(0x7f34dc0737d0 msgr2=0x7f34dc1524e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:45.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.010+0000 7f34e13b3700 1 -- 172.21.15.78:0/3613593099 shutdown_connections 2023-10-14T11:58:45.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.010+0000 7f34e13b3700 1 -- 172.21.15.78:0/3613593099 wait complete. 2023-10-14T11:58:45.015 INFO:teuthology.orchestra.run.smithi078.stderr:listed 0 entries 2023-10-14T11:58:45.024 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-14T11:58:45.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:45.025 DEBUG:teuthology.orchestra.run.smithi078:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2023-10-14T11:58:45.075 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi078.front.sepia.ceph.com 2023-10-14T11:58:45.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:45.076 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-14T11:58:45.101 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-14T11:58:45.102 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T11:58:45.124 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-14T11:58:45.124 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T11:58:45.184 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-14T11:58:45.184 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T11:58:45.245 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T11:58:45.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.363+0000 7f4303d6a700 1 Processor -- start 2023-10-14T11:58:45.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.363+0000 7f4303d6a700 1 -- start start 2023-10-14T11:58:45.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.364+0000 7f4303d6a700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f42fc06ab40 0x7f42fc0a9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:45.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.364+0000 7f4303d6a700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f42fc0c14c0 0x7f42fc155530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:45.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.364+0000 7f4303d6a700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42fc0a7e50 con 0x7f42fc06af20 2023-10-14T11:58:45.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.364+0000 7f4303d6a700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42fc0a80f0 con 0x7f42fc0c14c0 2023-10-14T11:58:45.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.364+0000 7f4303d6a700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42fc0c2510 con 0x7f42fc06ab40 2023-10-14T11:58:45.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.364+0000 7f4301305700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f42fc06af20 0x7f42fc0c2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50552/0 (socket says 172.21.15.78:50552) 2023-10-14T11:58:45.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.364+0000 7f4301305700 1 -- 172.21.15.78:0/2092470825 learned_addr learned my addr 172.21.15.78:0/2092470825 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:45.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.365+0000 7f4302307700 1 --2- 172.21.15.78:0/2092470825 >> v2:172.21.15.195:3300/0 conn(0x7f42fc0c14c0 0x7f42fc155530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:45.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.365+0000 7f4301b06700 1 --2- 172.21.15.78:0/2092470825 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f42fc06ab40 0x7f42fc0a9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:45.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.365+0000 7f4300b04700 1 -- 172.21.15.78:0/2092470825 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950586572 0 0) 0x7f42fc0a7e50 con 0x7f42fc06af20 2023-10-14T11:58:45.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.365+0000 7f4300b04700 1 -- 172.21.15.78:0/2092470825 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42e80035b0 con 0x7f42fc06af20 2023-10-14T11:58:45.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.365+0000 7f4300b04700 1 -- 172.21.15.78:0/2092470825 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 879132089 0 0) 0x7f42e80035b0 con 0x7f42fc06af20 2023-10-14T11:58:45.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.365+0000 7f4300b04700 1 -- 172.21.15.78:0/2092470825 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f42fc06ab40 msgr2=0x7f42fc0a9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:45.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.365+0000 7f4300b04700 1 --2- 172.21.15.78:0/2092470825 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f42fc06ab40 0x7f42fc0a9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:45.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.365+0000 7f4300b04700 1 -- 172.21.15.78:0/2092470825 >> v2:172.21.15.195:3300/0 conn(0x7f42fc0c14c0 msgr2=0x7f42fc155530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:45.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.365+0000 7f4300b04700 1 --2- 172.21.15.78:0/2092470825 >> v2:172.21.15.195:3300/0 conn(0x7f42fc0c14c0 0x7f42fc155530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:45.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.365+0000 7f4300b04700 1 -- 172.21.15.78:0/2092470825 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42fc0c3070 con 0x7f42fc06af20 2023-10-14T11:58:45.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.365+0000 7f4300b04700 1 -- 172.21.15.78:0/2092470825 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441799917 0 0) 0x7f42f8001670 con 0x7f42fc06af20 2023-10-14T11:58:45.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.365+0000 7f4300b04700 1 -- 172.21.15.78:0/2092470825 >> v1:172.21.15.78:6789/0 conn(0x7f42fc06af20 legacy=0x7f42fc0c2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:45.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.365+0000 7f4300b04700 1 --2- 172.21.15.78:0/2092470825 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f42fc06ab40 0x7f42e8003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:45.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.365+0000 7f4300b04700 1 --2- 172.21.15.78:0/2092470825 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f42e8003f80 0x7f42e8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:45.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.365+0000 7f4300b04700 1 --2- 172.21.15.78:0/2092470825 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42e8004d90 0x7f42e8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:45.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.365+0000 7f4300b04700 1 -- 172.21.15.78:0/2092470825 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42fc0a80f0 con 0x7f42e8004d90 2023-10-14T11:58:45.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.365+0000 7f4300b04700 1 -- 172.21.15.78:0/2092470825 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42fc0c2510 con 0x7f42e8003f80 2023-10-14T11:58:45.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.365+0000 7f4300b04700 1 -- 172.21.15.78:0/2092470825 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42e800dc70 con 0x7f42fc06ab40 2023-10-14T11:58:45.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.365+0000 7f4302307700 1 --2- 172.21.15.78:0/2092470825 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42e8004d90 0x7f42e8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:45.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.365+0000 7f4301b06700 1 --2- 172.21.15.78:0/2092470825 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f42fc06ab40 0x7f42e8003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:45.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.365+0000 7f4301305700 1 --2- 172.21.15.78:0/2092470825 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f42e8003f80 0x7f42e8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:45.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.365+0000 7f4302307700 1 -- 172.21.15.78:0/2092470825 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f42fc06ab40 msgr2=0x7f42e8003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:45.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.365+0000 7f4302307700 1 --2- 172.21.15.78:0/2092470825 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f42fc06ab40 0x7f42e8003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:45.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.365+0000 7f4302307700 1 -- 172.21.15.78:0/2092470825 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f42e8003f80 msgr2=0x7f42e8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:45.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.365+0000 7f4302307700 1 --2- 172.21.15.78:0/2092470825 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f42e8003f80 0x7f42e8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:45.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.365+0000 7f4302307700 1 -- 172.21.15.78:0/2092470825 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e80035b0 con 0x7f42e8004d90 2023-10-14T11:58:45.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.365+0000 7f4302307700 1 --2- 172.21.15.78:0/2092470825 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42e8004d90 0x7f42e8005210 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f42f400e170 tx=0x7f42e8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc4728b05666e2d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:45.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.366+0000 7f4300b04700 1 -- 172.21.15.78:0/2092470825 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f400ec90 con 0x7f42e8004d90 2023-10-14T11:58:45.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.366+0000 7f4300b04700 1 -- 172.21.15.78:0/2092470825 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42f400f040 con 0x7f42e8004d90 2023-10-14T11:58:45.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.366+0000 7f4300b04700 1 -- 172.21.15.78:0/2092470825 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f4014480 con 0x7f42e8004d90 2023-10-14T11:58:45.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.366+0000 7f4303d6a700 1 -- 172.21.15.78:0/2092470825 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42e8004d90 msgr2=0x7f42e8005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:45.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.366+0000 7f4303d6a700 1 --2- 172.21.15.78:0/2092470825 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42e8004d90 0x7f42e8005210 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f42f400e170 tx=0x7f42e8005880 comp rx=0 tx=0).stop 2023-10-14T11:58:45.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.366+0000 7f4302307700 1 -- 172.21.15.78:0/2092470825 reap_dead start 2023-10-14T11:58:45.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.366+0000 7f4303d6a700 1 -- 172.21.15.78:0/2092470825 shutdown_connections 2023-10-14T11:58:45.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.366+0000 7f4303d6a700 1 -- 172.21.15.78:0/2092470825 >> 172.21.15.78:0/2092470825 conn(0x7f42fc0a8410 msgr2=0x7f42fc0a8820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:45.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.366+0000 7f4303d6a700 1 -- 172.21.15.78:0/2092470825 shutdown_connections 2023-10-14T11:58:45.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.366+0000 7f4303d6a700 1 -- 172.21.15.78:0/2092470825 wait complete. 2023-10-14T11:58:45.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.367+0000 7f4303d6a700 1 Processor -- start 2023-10-14T11:58:45.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.367+0000 7f4303d6a700 1 -- start start 2023-10-14T11:58:45.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.368+0000 7f4303d6a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f42fc1499b0 0x7f42fc1ea770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:45.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.368+0000 7f4303d6a700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f42fc0bd8f0 0x7f42fc14fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:45.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.368+0000 7f4303d6a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42fc150060 0x7f42fc153100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:45.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.368+0000 7f4303d6a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42fc152990 con 0x7f42fc150060 2023-10-14T11:58:45.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.368+0000 7f4303d6a700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42fc152c70 con 0x7f42fc1499b0 2023-10-14T11:58:45.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.368+0000 7f4303d6a700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42fc0c60d0 con 0x7f42fc0bd8f0 2023-10-14T11:58:45.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.368+0000 7f4302307700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42fc150060 0x7f42fc153100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:45.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.368+0000 7f4302307700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42fc150060 0x7f42fc153100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45690/0 (socket says 172.21.15.78:45690) 2023-10-14T11:58:45.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.368+0000 7f4302307700 1 -- 172.21.15.78:0/3524055076 learned_addr learned my addr 172.21.15.78:0/3524055076 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:45.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.368+0000 7f4301305700 1 --2- 172.21.15.78:0/3524055076 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f42fc0bd8f0 0x7f42fc14fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:45.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.368+0000 7f4301b06700 1 --2- 172.21.15.78:0/3524055076 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f42fc1499b0 0x7f42fc1ea770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:45.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.368+0000 7f4302307700 1 -- 172.21.15.78:0/3524055076 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f42fc0bd8f0 msgr2=0x7f42fc14fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:45.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.368+0000 7f4302307700 1 --2- 172.21.15.78:0/3524055076 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f42fc0bd8f0 0x7f42fc14fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:45.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.368+0000 7f4302307700 1 -- 172.21.15.78:0/3524055076 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f42fc1499b0 msgr2=0x7f42fc1ea770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:45.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.368+0000 7f4302307700 1 --2- 172.21.15.78:0/3524055076 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f42fc1499b0 0x7f42fc1ea770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:45.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.368+0000 7f4302307700 1 -- 172.21.15.78:0/3524055076 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42fc0c63b0 con 0x7f42fc150060 2023-10-14T11:58:45.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.368+0000 7f4302307700 1 --2- 172.21.15.78:0/3524055076 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42fc150060 0x7f42fc153100 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f42f4016000 tx=0x7f42f4004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:45.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.369+0000 7f42f27fc700 1 -- 172.21.15.78:0/3524055076 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f4007000 con 0x7f42fc150060 2023-10-14T11:58:45.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.369+0000 7f42f27fc700 1 -- 172.21.15.78:0/3524055076 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42f400ee80 con 0x7f42fc150060 2023-10-14T11:58:45.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.369+0000 7f42f27fc700 1 -- 172.21.15.78:0/3524055076 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f4003a20 con 0x7f42fc150060 2023-10-14T11:58:45.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.369+0000 7f4303d6a700 1 -- 172.21.15.78:0/3524055076 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42fc071380 con 0x7f42fc150060 2023-10-14T11:58:45.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.369+0000 7f4303d6a700 1 -- 172.21.15.78:0/3524055076 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42fc149f70 con 0x7f42fc150060 2023-10-14T11:58:45.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.369+0000 7f4303d6a700 1 -- 172.21.15.78:0/3524055076 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42fc0c2380 con 0x7f42fc150060 2023-10-14T11:58:45.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.370+0000 7f42f27fc700 1 -- 172.21.15.78:0/3524055076 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f42f4003c00 con 0x7f42fc150060 2023-10-14T11:58:45.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.370+0000 7f42f27fc700 1 --2- 172.21.15.78:0/3524055076 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f42e006af00 0x7f42e006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:45.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.371+0000 7f42f27fc700 1 -- 172.21.15.78:0/3524055076 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f42f408a6d0 con 0x7f42fc150060 2023-10-14T11:58:45.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.371+0000 7f4301b06700 1 --2- 172.21.15.78:0/3524055076 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f42e006af00 0x7f42e006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:45.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.371+0000 7f4301b06700 1 --2- 172.21.15.78:0/3524055076 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f42e006af00 0x7f42e006d3c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f42ec009ba0 tx=0x7f42ec008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:45.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.375+0000 7f42f27fc700 1 -- 172.21.15.78:0/3524055076 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f42f4055810 con 0x7f42fc150060 2023-10-14T11:58:45.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.567+0000 7f4303d6a700 1 -- 172.21.15.78:0/3524055076 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42fc0c6540 con 0x7f42fc150060 2023-10-14T11:58:45.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.568+0000 7f42f27fc700 1 -- 172.21.15.78:0/3524055076 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16481 (secure 0 0 0) 0x7f42f4018090 con 0x7f42fc150060 2023-10-14T11:58:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:45.571 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":21,"fsid":"4c5dc134-3092-406a-a1e1-13e6d9e926e8","created":"2023-10-14T11:57:57.363750+0000","modified":"2023-10-14T11:58:16.139413+0000","last_up_change":"2023-10-14T11:58:06.982472+0000","last_in_change":"2023-10-14T11:57:59.821373+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T11:58:09.001200+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T11:58:12.778573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T11:58:13.490286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"625a5fc0-9474-4eae-b80e-14c27a74ad19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6801","nonce":3884969780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6803","nonce":3884969780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6807","nonce":3884969780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6805","nonce":3884969780}]},"public_addr":"172.21.15.78:6801/3884969780","cluster_addr":"172.21.15.78:6803/3884969780","heartbeat_back_addr":"172.21.15.78:6807/3884969780","heartbeat_front_addr":"172.21.15.78:6805/3884969780","state":["exists","up"]},{"osd":1,"uuid":"49d63db1-2981-49ab-9c40-c50fc744b982","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6809","nonce":2176928406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6811","nonce":2176928406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6815","nonce":2176928406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6813","nonce":2176928406}]},"public_addr":"172.21.15.78:6809/2176928406","cluster_addr":"172.21.15.78:6811/2176928406","heartbeat_back_addr":"172.21.15.78:6815/2176928406","heartbeat_front_addr":"172.21.15.78:6813/2176928406","state":["exists","up"]},{"osd":2,"uuid":"1d7d0b29-cbd1-4717-9fe6-765a8d4998b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6825","nonce":3647044577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6827","nonce":3647044577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6831","nonce":3647044577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6829","nonce":3647044577}]},"public_addr":"172.21.15.78:6825/3647044577","cluster_addr":"172.21.15.78:6827/3647044577","heartbeat_back_addr":"172.21.15.78:6831/3647044577","heartbeat_front_addr":"172.21.15.78:6829/3647044577","state":["exists","up"]},{"osd":3,"uuid":"2f75ded1-e9e2-40e7-99ae-45e1e425e1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6817","nonce":3584112288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6819","nonce":3584112288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6823","nonce":3584112288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6821","nonce":3584112288}]},"public_addr":"172.21.15.78:6817/3584112288","cluster_addr":"172.21.15.78:6819/3584112288","heartbeat_back_addr":"172.21.15.78:6823/3584112288","heartbeat_front_addr":"172.21.15.78:6821/3584112288","state":["exists","up"]},{"osd":4,"uuid":"5130b432-8358-46cb-8a31-3064c976f6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6801","nonce":3732756301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6803","nonce":3732756301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6807","nonce":3732756301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6805","nonce":3732756301}]},"public_addr":"172.21.15.195:6801/3732756301","cluster_addr":"172.21.15.195:6803/3732756301","heartbeat_back_addr":"172.21.15.195:6807/3732756301","heartbeat_front_addr":"172.21.15.195:6805/3732756301","state":["exists","up"]},{"osd":5,"uuid":"60e234a9-a506-4e40-9c19-49a58f3cf28c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6825","nonce":2857706411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6827","nonce":2857706411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6831","nonce":2857706411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6829","nonce":2857706411}]},"public_addr":"172.21.15.195:6825/2857706411","cluster_addr":"172.21.15.195:6827/2857706411","heartbeat_back_addr":"172.21.15.195:6831/2857706411","heartbeat_front_addr":"172.21.15.195:6829/2857706411","state":["exists","up"]},{"osd":6,"uuid":"df7d2dd6-26b6-4eb1-96e5-94d1f3955b33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6810","nonce":72554374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6813","nonce":72554374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6820","nonce":72554374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6817","nonce":72554374}]},"public_addr":"172.21.15.195:6810/72554374","cluster_addr":"172.21.15.195:6813/72554374","heartbeat_back_addr":"172.21.15.195:6820/72554374","heartbeat_front_addr":"172.21.15.195:6817/72554374","state":["exists","up"]},{"osd":7,"uuid":"47a9d4de-f56d-4aa4-9900-c4a9cf9b2f97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6809","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6811","nonce":482472979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6816","nonce":482472979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6823","nonce":482472979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6821","nonce":482472979}]},"public_addr":"172.21.15.195:6811/482472979","cluster_addr":"172.21.15.195:6816/482472979","heartbeat_back_addr":"172.21.15.195:6823/482472979","heartbeat_front_addr":"172.21.15.195:6821/482472979","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.754003+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.005994+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.043603+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.024506+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.062585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.033767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.994825+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.038300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T11:58:45.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.570+0000 7f4303d6a700 1 -- 172.21.15.78:0/3524055076 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f42e006af00 msgr2=0x7f42e006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:45.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.570+0000 7f4303d6a700 1 --2- 172.21.15.78:0/3524055076 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f42e006af00 0x7f42e006d3c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f42ec009ba0 tx=0x7f42ec008040 comp rx=0 tx=0).stop 2023-10-14T11:58:45.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.570+0000 7f4303d6a700 1 -- 172.21.15.78:0/3524055076 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42fc150060 msgr2=0x7f42fc153100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:45.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.570+0000 7f4303d6a700 1 --2- 172.21.15.78:0/3524055076 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42fc150060 0x7f42fc153100 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f42f4016000 tx=0x7f42f4004b40 comp rx=0 tx=0).stop 2023-10-14T11:58:45.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.571+0000 7f4303d6a700 1 -- 172.21.15.78:0/3524055076 shutdown_connections 2023-10-14T11:58:45.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.571+0000 7f4303d6a700 1 --2- 172.21.15.78:0/3524055076 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42fc150060 0x7f42fc153100 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:45.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.571+0000 7f4303d6a700 1 --2- 172.21.15.78:0/3524055076 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f42e006af00 0x7f42e006d3c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:45.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.571+0000 7f4303d6a700 1 --2- 172.21.15.78:0/3524055076 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f42fc1499b0 0x7f42fc1ea770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:45.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.571+0000 7f4303d6a700 1 --2- 172.21.15.78:0/3524055076 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f42fc0bd8f0 0x7f42fc14fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:45.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.571+0000 7f4303d6a700 1 -- 172.21.15.78:0/3524055076 >> 172.21.15.78:0/3524055076 conn(0x7f42fc0a8410 msgr2=0x7f42fc0a9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:45.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.571+0000 7f4303d6a700 1 -- 172.21.15.78:0/3524055076 shutdown_connections 2023-10-14T11:58:45.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.571+0000 7f4303d6a700 1 -- 172.21.15.78:0/3524055076 wait complete. 2023-10-14T11:58:45.585 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T11:58:09.001200+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T11:58:12.778573+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T11:58:13.490286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T11:58:45.585 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T11:58:45.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.706+0000 7fd2e3ac2700 1 Processor -- start 2023-10-14T11:58:45.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.706+0000 7fd2e3ac2700 1 -- start start 2023-10-14T11:58:45.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.707+0000 7fd2e3ac2700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd2dc0a70c0 0x7fd2dc154960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:45.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.707+0000 7fd2e3ac2700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd2dc154ea0 0x7fd2dc15d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:45.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.707+0000 7fd2e3ac2700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2dc1537b0 con 0x7fd2dc1533d0 2023-10-14T11:58:45.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.707+0000 7fd2e3ac2700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2dc0c6fb0 con 0x7fd2dc154ea0 2023-10-14T11:58:45.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.707+0000 7fd2e3ac2700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2dc0c7f30 con 0x7fd2dc0a70c0 2023-10-14T11:58:45.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.707+0000 7fd2e205f700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd2dc154ea0 0x7fd2dc15d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:45.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.707+0000 7fd2e205f700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd2dc154ea0 0x7fd2dc15d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45698/0 (socket says 172.21.15.78:45698) 2023-10-14T11:58:45.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.707+0000 7fd2e205f700 1 -- 172.21.15.78:0/494072768 learned_addr learned my addr 172.21.15.78:0/494072768 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:45.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.707+0000 7fd2e105d700 1 --2- 172.21.15.78:0/494072768 >> v2:172.21.15.195:3300/0 conn(0x7fd2dc0a70c0 0x7fd2dc154960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:45.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.707+0000 7fd2e205f700 1 -- 172.21.15.78:0/494072768 >> v2:172.21.15.195:3300/0 conn(0x7fd2dc0a70c0 msgr2=0x7fd2dc154960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:45.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.707+0000 7fd2e205f700 1 --2- 172.21.15.78:0/494072768 >> v2:172.21.15.195:3300/0 conn(0x7fd2dc0a70c0 0x7fd2dc154960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:45.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.707+0000 7fd2e205f700 1 -- 172.21.15.78:0/494072768 >> v1:172.21.15.195:6789/0 conn(0x7fd2dc1533d0 legacy=0x7fd2dc0c8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:45.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.707+0000 7fd2e205f700 1 -- 172.21.15.78:0/494072768 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2dc0c7c50 con 0x7fd2dc154ea0 2023-10-14T11:58:45.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.707+0000 7fd2e205f700 1 --2- 172.21.15.78:0/494072768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2dc154ea0 0x7fd2dc15d130 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc0c5c30 tx=0x7fd2d400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37cfa24e6213d2bd server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:45.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.707+0000 7fd2e085c700 1 -- 172.21.15.78:0/494072768 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d400e070 con 0x7fd2dc154ea0 2023-10-14T11:58:45.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.707+0000 7fd2e085c700 1 -- 172.21.15.78:0/494072768 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d4003a20 con 0x7fd2dc154ea0 2023-10-14T11:58:45.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.707+0000 7fd2e085c700 1 -- 172.21.15.78:0/494072768 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d4003dd0 con 0x7fd2dc154ea0 2023-10-14T11:58:45.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.708+0000 7fd2e3ac2700 1 -- 172.21.15.78:0/494072768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2dc154ea0 msgr2=0x7fd2dc15d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:45.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.708+0000 7fd2e3ac2700 1 --2- 172.21.15.78:0/494072768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2dc154ea0 0x7fd2dc15d130 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc0c5c30 tx=0x7fd2d400b330 comp rx=0 tx=0).stop 2023-10-14T11:58:45.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.708+0000 7fd2e3ac2700 1 -- 172.21.15.78:0/494072768 shutdown_connections 2023-10-14T11:58:45.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.708+0000 7fd2e3ac2700 1 --2- 172.21.15.78:0/494072768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2dc154ea0 0x7fd2dc15d130 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:45.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.708+0000 7fd2e3ac2700 1 --2- 172.21.15.78:0/494072768 >> v2:172.21.15.195:3300/0 conn(0x7fd2dc0a70c0 0x7fd2dc154960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:45.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.708+0000 7fd2e3ac2700 1 -- 172.21.15.78:0/494072768 >> 172.21.15.78:0/494072768 conn(0x7fd2dc0737d0 msgr2=0x7fd2dc073be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:45.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.708+0000 7fd2e3ac2700 1 -- 172.21.15.78:0/494072768 shutdown_connections 2023-10-14T11:58:45.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.708+0000 7fd2e3ac2700 1 -- 172.21.15.78:0/494072768 wait complete. 2023-10-14T11:58:45.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.709+0000 7fd2e3ac2700 1 Processor -- start 2023-10-14T11:58:45.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.709+0000 7fd2e3ac2700 1 -- start start 2023-10-14T11:58:45.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.710+0000 7fd2e3ac2700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd2dc0a70c0 0x7fd2dc0bbc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:45.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.710+0000 7fd2e3ac2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2dc1533d0 0x7fd2dc0ac0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:45.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.710+0000 7fd2e3ac2700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd2dc154ea0 0x7fd2dc1ed020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:45.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.710+0000 7fd2e3ac2700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2dc0c7a70 con 0x7fd2dc1533d0 2023-10-14T11:58:45.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.710+0000 7fd2e3ac2700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2dc0acee0 con 0x7fd2dc0a70c0 2023-10-14T11:58:45.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.710+0000 7fd2e3ac2700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2dc1edd60 con 0x7fd2dc154ea0 2023-10-14T11:58:45.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.710+0000 7fd2e105d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2dc1533d0 0x7fd2dc0ac0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:45.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.710+0000 7fd2e105d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2dc1533d0 0x7fd2dc0ac0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45702/0 (socket says 172.21.15.78:45702) 2023-10-14T11:58:45.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.710+0000 7fd2e105d700 1 -- 172.21.15.78:0/3309747455 learned_addr learned my addr 172.21.15.78:0/3309747455 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:45.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.710+0000 7fd2e185e700 1 --2- 172.21.15.78:0/3309747455 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd2dc0a70c0 0x7fd2dc0bbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:45.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.710+0000 7fd2e205f700 1 --2- 172.21.15.78:0/3309747455 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd2dc154ea0 0x7fd2dc1ed020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:45.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.710+0000 7fd2e105d700 1 -- 172.21.15.78:0/3309747455 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd2dc154ea0 msgr2=0x7fd2dc1ed020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:45.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.710+0000 7fd2e105d700 1 --2- 172.21.15.78:0/3309747455 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd2dc154ea0 0x7fd2dc1ed020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:45.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.710+0000 7fd2e105d700 1 -- 172.21.15.78:0/3309747455 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd2dc0a70c0 msgr2=0x7fd2dc0bbc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:45.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.710+0000 7fd2e105d700 1 --2- 172.21.15.78:0/3309747455 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd2dc0a70c0 0x7fd2dc0bbc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:45.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.710+0000 7fd2e105d700 1 -- 172.21.15.78:0/3309747455 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d4009b30 con 0x7fd2dc1533d0 2023-10-14T11:58:45.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.710+0000 7fd2e205f700 1 --2- 172.21.15.78:0/3309747455 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd2dc154ea0 0x7fd2dc1ed020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:45.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.711+0000 7fd2e105d700 1 --2- 172.21.15.78:0/3309747455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2dc1533d0 0x7fd2dc0ac0f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc00d800 tx=0x7fd2cc00db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:45.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.713+0000 7fd2d27fc700 1 -- 172.21.15.78:0/3309747455 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc0125f0 con 0x7fd2dc1533d0 2023-10-14T11:58:45.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.713+0000 7fd2d27fc700 1 -- 172.21.15.78:0/3309747455 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2cc012750 con 0x7fd2dc1533d0 2023-10-14T11:58:45.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.713+0000 7fd2e3ac2700 1 -- 172.21.15.78:0/3309747455 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2dc0bc1d0 con 0x7fd2dc1533d0 2023-10-14T11:58:45.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.713+0000 7fd2d27fc700 1 -- 172.21.15.78:0/3309747455 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc012b00 con 0x7fd2dc1533d0 2023-10-14T11:58:45.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.713+0000 7fd2e3ac2700 1 -- 172.21.15.78:0/3309747455 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2dc0abbd0 con 0x7fd2dc1533d0 2023-10-14T11:58:45.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.714+0000 7fd2e3ac2700 1 -- 172.21.15.78:0/3309747455 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2dc0abdf0 con 0x7fd2dc1533d0 2023-10-14T11:58:45.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.714+0000 7fd2d27fc700 1 -- 172.21.15.78:0/3309747455 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd2cc012c60 con 0x7fd2dc1533d0 2023-10-14T11:58:45.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.714+0000 7fd2d27fc700 1 --2- 172.21.15.78:0/3309747455 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd2c006af00 0x7fd2c006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.715+0000 7fd2d27fc700 1 -- 172.21.15.78:0/3309747455 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd2cc0898e0 con 0x7fd2dc1533d0 2023-10-14T11:58:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.715+0000 7fd2e185e700 1 --2- 172.21.15.78:0/3309747455 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd2c006af00 0x7fd2c006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.715+0000 7fd2e185e700 1 --2- 172.21.15.78:0/3309747455 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd2c006af00 0x7fd2c006d3c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8007fd0 tx=0x7fd2d8007ec0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.719+0000 7fd2d27fc700 1 -- 172.21.15.78:0/3309747455 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd2cc0549a0 con 0x7fd2dc1533d0 2023-10-14T11:58:45.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.907+0000 7fd2e3ac2700 1 -- 172.21.15.78:0/3309747455 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd2dc1edb90 con 0x7fd2dc1533d0 2023-10-14T11:58:45.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.907+0000 7fd2d27fc700 1 -- 172.21.15.78:0/3309747455 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd2cc057fc0 con 0x7fd2dc1533d0 2023-10-14T11:58:45.910 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-10-14T11:58:45.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.909+0000 7fd2e3ac2700 1 -- 172.21.15.78:0/3309747455 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd2c006af00 msgr2=0x7fd2c006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:45.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.909+0000 7fd2e3ac2700 1 --2- 172.21.15.78:0/3309747455 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd2c006af00 0x7fd2c006d3c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8007fd0 tx=0x7fd2d8007ec0 comp rx=0 tx=0).stop 2023-10-14T11:58:45.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.909+0000 7fd2e3ac2700 1 -- 172.21.15.78:0/3309747455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2dc1533d0 msgr2=0x7fd2dc0ac0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:45.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.909+0000 7fd2e3ac2700 1 --2- 172.21.15.78:0/3309747455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2dc1533d0 0x7fd2dc0ac0f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc00d800 tx=0x7fd2cc00db10 comp rx=0 tx=0).stop 2023-10-14T11:58:45.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.909+0000 7fd2e3ac2700 1 -- 172.21.15.78:0/3309747455 shutdown_connections 2023-10-14T11:58:45.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.909+0000 7fd2e3ac2700 1 --2- 172.21.15.78:0/3309747455 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd2dc154ea0 0x7fd2dc1ed020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:45.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.909+0000 7fd2e3ac2700 1 --2- 172.21.15.78:0/3309747455 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd2c006af00 0x7fd2c006d3c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:45.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.909+0000 7fd2e3ac2700 1 --2- 172.21.15.78:0/3309747455 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd2dc0a70c0 0x7fd2dc0bbc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:45.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.909+0000 7fd2e3ac2700 1 --2- 172.21.15.78:0/3309747455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2dc1533d0 0x7fd2dc0ac0f0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:45.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.909+0000 7fd2e3ac2700 1 -- 172.21.15.78:0/3309747455 >> 172.21.15.78:0/3309747455 conn(0x7fd2dc0737d0 msgr2=0x7fd2dc0c85d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:45.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.909+0000 7fd2e3ac2700 1 -- 172.21.15.78:0/3309747455 shutdown_connections 2023-10-14T11:58:45.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:45.909+0000 7fd2e3ac2700 1 -- 172.21.15.78:0/3309747455 wait complete. 2023-10-14T11:58:45.923 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T11:58:46.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.044+0000 7f148b3dd700 1 Processor -- start 2023-10-14T11:58:46.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.044+0000 7f148b3dd700 1 -- start start 2023-10-14T11:58:46.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.045+0000 7f148b3dd700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f148414e6f0 0x7f148415c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:46.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.045+0000 7f148b3dd700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14841542e0 con 0x7f14840c11a0 2023-10-14T11:58:46.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.045+0000 7f148b3dd700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14841519f0 con 0x7f14840c69e0 2023-10-14T11:58:46.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.045+0000 7f148b3dd700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1484151f20 con 0x7f148414e6f0 2023-10-14T11:58:46.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.045+0000 7f1488978700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f14840c11a0 0x7f148414dfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50584/0 (socket says 172.21.15.78:50584) 2023-10-14T11:58:46.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.045+0000 7f1488978700 1 -- 172.21.15.78:0/1745713332 learned_addr learned my addr 172.21.15.78:0/1745713332 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:46.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.045+0000 7f148997a700 1 --2- 172.21.15.78:0/1745713332 >> v2:172.21.15.78:3300/0 conn(0x7f148414e6f0 0x7f148415c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:46.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.045+0000 7f147bfff700 1 -- 172.21.15.78:0/1745713332 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2870229101 0 0) 0x7f14841542e0 con 0x7f14840c11a0 2023-10-14T11:58:46.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.045+0000 7f147bfff700 1 -- 172.21.15.78:0/1745713332 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14700035b0 con 0x7f14840c11a0 2023-10-14T11:58:46.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.045+0000 7f148997a700 1 -- 172.21.15.78:0/1745713332 >> v1:172.21.15.195:6789/0 conn(0x7f14840c69e0 legacy=0x7f148406e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:46.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.045+0000 7f148997a700 1 -- 172.21.15.78:0/1745713332 >> v1:172.21.15.78:6789/0 conn(0x7f14840c11a0 legacy=0x7f148414dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:46.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.045+0000 7f148997a700 1 -- 172.21.15.78:0/1745713332 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1484070c50 con 0x7f148414e6f0 2023-10-14T11:58:46.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.045+0000 7f148997a700 1 --2- 172.21.15.78:0/1745713332 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f148414e6f0 0x7f148415c820 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f147c00aea0 tx=0x7f147c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a0f96f4a2c27536 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:46.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.046+0000 7f147bfff700 1 -- 172.21.15.78:0/1745713332 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f147c00a2d0 con 0x7f148414e6f0 2023-10-14T11:58:46.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.046+0000 7f147bfff700 1 -- 172.21.15.78:0/1745713332 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f147c00c910 con 0x7f148414e6f0 2023-10-14T11:58:46.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.046+0000 7f147bfff700 1 -- 172.21.15.78:0/1745713332 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f147c00cca0 con 0x7f148414e6f0 2023-10-14T11:58:46.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.046+0000 7f148b3dd700 1 -- 172.21.15.78:0/1745713332 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f148414e6f0 msgr2=0x7f148415c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:46.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.046+0000 7f148b3dd700 1 --2- 172.21.15.78:0/1745713332 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f148414e6f0 0x7f148415c820 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f147c00aea0 tx=0x7f147c00c330 comp rx=0 tx=0).stop 2023-10-14T11:58:46.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.046+0000 7f148b3dd700 1 -- 172.21.15.78:0/1745713332 shutdown_connections 2023-10-14T11:58:46.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.046+0000 7f148b3dd700 1 --2- 172.21.15.78:0/1745713332 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f148414e6f0 0x7f148415c820 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:46.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.046+0000 7f148b3dd700 1 -- 172.21.15.78:0/1745713332 >> 172.21.15.78:0/1745713332 conn(0x7f14840a8410 msgr2=0x7f14840a8820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:46.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.046+0000 7f148b3dd700 1 -- 172.21.15.78:0/1745713332 shutdown_connections 2023-10-14T11:58:46.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.046+0000 7f148b3dd700 1 -- 172.21.15.78:0/1745713332 wait complete. 2023-10-14T11:58:46.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.047+0000 7f148b3dd700 1 Processor -- start 2023-10-14T11:58:46.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.047+0000 7f148b3dd700 1 -- start start 2023-10-14T11:58:46.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.048+0000 7f148b3dd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14840c11a0 0x7f14840ba5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:46.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.048+0000 7f148b3dd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f14840c69e0 0x7f14840bab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:46.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.048+0000 7f148b3dd700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f148414e6f0 0x7f14840bdb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:46.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.048+0000 7f148b3dd700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14840be880 con 0x7f14840c11a0 2023-10-14T11:58:46.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.048+0000 7f148b3dd700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14840c0820 con 0x7f14840c69e0 2023-10-14T11:58:46.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.048+0000 7f148b3dd700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14840c0080 con 0x7f148414e6f0 2023-10-14T11:58:46.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.048+0000 7f1489179700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14840c11a0 0x7f14840ba5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:46.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.048+0000 7f1489179700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14840c11a0 0x7f14840ba5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45712/0 (socket says 172.21.15.78:45712) 2023-10-14T11:58:46.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.048+0000 7f1489179700 1 -- 172.21.15.78:0/2522894651 learned_addr learned my addr 172.21.15.78:0/2522894651 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:46.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.048+0000 7f1488978700 1 --2- 172.21.15.78:0/2522894651 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f14840c69e0 0x7f14840bab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:46.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.048+0000 7f148997a700 1 --2- 172.21.15.78:0/2522894651 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f148414e6f0 0x7f14840bdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:46.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.048+0000 7f1489179700 1 -- 172.21.15.78:0/2522894651 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f148414e6f0 msgr2=0x7f14840bdb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:46.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.048+0000 7f1489179700 1 --2- 172.21.15.78:0/2522894651 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f148414e6f0 0x7f14840bdb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:46.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.048+0000 7f1489179700 1 -- 172.21.15.78:0/2522894651 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f14840c69e0 msgr2=0x7f14840bab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:46.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.048+0000 7f1489179700 1 --2- 172.21.15.78:0/2522894651 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f14840c69e0 0x7f14840bab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:46.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.048+0000 7f1489179700 1 -- 172.21.15.78:0/2522894651 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f147c00ab80 con 0x7f14840c11a0 2023-10-14T11:58:46.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.049+0000 7f1489179700 1 --2- 172.21.15.78:0/2522894651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14840c11a0 0x7f14840ba5c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f147400c510 tx=0x7f147400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:46.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.049+0000 7f1479ffb700 1 -- 172.21.15.78:0/2522894651 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14740052d0 con 0x7f14840c11a0 2023-10-14T11:58:46.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.049+0000 7f1479ffb700 1 -- 172.21.15.78:0/2522894651 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1474005430 con 0x7f14840c11a0 2023-10-14T11:58:46.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.049+0000 7f148b3dd700 1 -- 172.21.15.78:0/2522894651 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14841daf40 con 0x7f14840c11a0 2023-10-14T11:58:46.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.049+0000 7f1479ffb700 1 -- 172.21.15.78:0/2522894651 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14740038c0 con 0x7f14840c11a0 2023-10-14T11:58:46.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.049+0000 7f148b3dd700 1 -- 172.21.15.78:0/2522894651 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14841db220 con 0x7f14840c11a0 2023-10-14T11:58:46.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.050+0000 7f148b3dd700 1 -- 172.21.15.78:0/2522894651 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1484074b80 con 0x7f14840c11a0 2023-10-14T11:58:46.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.050+0000 7f1479ffb700 1 -- 172.21.15.78:0/2522894651 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1474003a20 con 0x7f14840c11a0 2023-10-14T11:58:46.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.051+0000 7f1479ffb700 1 --2- 172.21.15.78:0/2522894651 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f146806af00 0x7f146806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:46.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.051+0000 7f1479ffb700 1 -- 172.21.15.78:0/2522894651 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1474088a10 con 0x7f14840c11a0 2023-10-14T11:58:46.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.051+0000 7f1488978700 1 --2- 172.21.15.78:0/2522894651 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f146806af00 0x7f146806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:46.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.051+0000 7f1488978700 1 --2- 172.21.15.78:0/2522894651 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f146806af00 0x7f146806d3c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f14800011e0 tx=0x7f1480009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:46.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.055+0000 7f1479ffb700 1 -- 172.21.15.78:0/2522894651 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1474053b50 con 0x7f14840c11a0 2023-10-14T11:58:46.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.247+0000 7f148b3dd700 1 -- 172.21.15.78:0/2522894651 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f14840733e0 con 0x7f14840c11a0 2023-10-14T11:58:46.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.247+0000 7f1479ffb700 1 -- 172.21.15.78:0/2522894651 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1474017020 con 0x7f14840c11a0 2023-10-14T11:58:46.249 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-10-14T11:58:46.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.249+0000 7f148b3dd700 1 -- 172.21.15.78:0/2522894651 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f146806af00 msgr2=0x7f146806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:46.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.249+0000 7f148b3dd700 1 --2- 172.21.15.78:0/2522894651 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f146806af00 0x7f146806d3c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f14800011e0 tx=0x7f1480009040 comp rx=0 tx=0).stop 2023-10-14T11:58:46.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.249+0000 7f148b3dd700 1 -- 172.21.15.78:0/2522894651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14840c11a0 msgr2=0x7f14840ba5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:46.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.249+0000 7f148b3dd700 1 --2- 172.21.15.78:0/2522894651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14840c11a0 0x7f14840ba5c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f147400c510 tx=0x7f147400c8d0 comp rx=0 tx=0).stop 2023-10-14T11:58:46.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.249+0000 7f148b3dd700 1 -- 172.21.15.78:0/2522894651 shutdown_connections 2023-10-14T11:58:46.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.249+0000 7f148b3dd700 1 --2- 172.21.15.78:0/2522894651 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f148414e6f0 0x7f14840bdb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:46.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.249+0000 7f148b3dd700 1 --2- 172.21.15.78:0/2522894651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14840c11a0 0x7f14840ba5c0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:46.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.249+0000 7f148b3dd700 1 --2- 172.21.15.78:0/2522894651 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f146806af00 0x7f146806d3c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:46.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.249+0000 7f148b3dd700 1 --2- 172.21.15.78:0/2522894651 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f14840c69e0 0x7f14840bab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:46.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.249+0000 7f148b3dd700 1 -- 172.21.15.78:0/2522894651 >> 172.21.15.78:0/2522894651 conn(0x7f14840a8410 msgr2=0x7f14840c5ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:46.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.249+0000 7f148b3dd700 1 -- 172.21.15.78:0/2522894651 shutdown_connections 2023-10-14T11:58:46.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.249+0000 7f148b3dd700 1 -- 172.21.15.78:0/2522894651 wait complete. 2023-10-14T11:58:46.263 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T11:58:46.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.384+0000 7f7913ee9700 1 Processor -- start 2023-10-14T11:58:46.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.384+0000 7f7913ee9700 1 -- start start 2023-10-14T11:58:46.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.385+0000 7f7913ee9700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f790c158a30 0x7f790c13d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:46.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.385+0000 7f7913ee9700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f790c06b630 con 0x7f790c06ab40 2023-10-14T11:58:46.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.385+0000 7f7913ee9700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f790c0728a0 con 0x7f790c06f1c0 2023-10-14T11:58:46.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.385+0000 7f7913ee9700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f790c158e60 con 0x7f790c158a30 2023-10-14T11:58:46.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.385+0000 7f7912486700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f790c158a30 0x7f790c13d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:46.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.385+0000 7f7911c85700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f790c06ab40 0x7f790c06af20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50596/0 (socket says 172.21.15.78:50596) 2023-10-14T11:58:46.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.385+0000 7f7911c85700 1 -- 172.21.15.78:0/692332690 learned_addr learned my addr 172.21.15.78:0/692332690 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:46.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.385+0000 7f7912486700 1 -- 172.21.15.78:0/692332690 >> v1:172.21.15.195:6789/0 conn(0x7f790c06f1c0 legacy=0x7f790c154890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:46.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.385+0000 7f7912486700 1 -- 172.21.15.78:0/692332690 >> v1:172.21.15.78:6789/0 conn(0x7f790c06ab40 legacy=0x7f790c06af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:46.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.385+0000 7f7912486700 1 -- 172.21.15.78:0/692332690 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f790c157f90 con 0x7f790c158a30 2023-10-14T11:58:46.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.385+0000 7f7910c83700 1 -- 172.21.15.78:0/692332690 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1636670520 0 0) 0x7f790c06b630 con 0x7f790c06ab40 2023-10-14T11:58:46.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.385+0000 7f7912486700 1 --2- 172.21.15.78:0/692332690 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f790c158a30 0x7f790c13d840 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7904009d90 tx=0x7f790400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54105b292b61b1ad server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:46.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.386+0000 7f7910c83700 1 -- 172.21.15.78:0/692332690 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f790400e070 con 0x7f790c158a30 2023-10-14T11:58:46.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.386+0000 7f7910c83700 1 -- 172.21.15.78:0/692332690 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f790400bb90 con 0x7f790c158a30 2023-10-14T11:58:46.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.386+0000 7f7910c83700 1 -- 172.21.15.78:0/692332690 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7904003a20 con 0x7f790c158a30 2023-10-14T11:58:46.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.386+0000 7f7913ee9700 1 -- 172.21.15.78:0/692332690 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f790c158a30 msgr2=0x7f790c13d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:46.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.386+0000 7f7913ee9700 1 --2- 172.21.15.78:0/692332690 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f790c158a30 0x7f790c13d840 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7904009d90 tx=0x7f790400b330 comp rx=0 tx=0).stop 2023-10-14T11:58:46.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.386+0000 7f7913ee9700 1 -- 172.21.15.78:0/692332690 shutdown_connections 2023-10-14T11:58:46.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.386+0000 7f7913ee9700 1 --2- 172.21.15.78:0/692332690 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f790c158a30 0x7f790c13d840 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:46.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.386+0000 7f7913ee9700 1 -- 172.21.15.78:0/692332690 >> 172.21.15.78:0/692332690 conn(0x7f790c0c39c0 msgr2=0x7f790c0c3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:46.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.386+0000 7f7913ee9700 1 -- 172.21.15.78:0/692332690 shutdown_connections 2023-10-14T11:58:46.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.386+0000 7f7913ee9700 1 -- 172.21.15.78:0/692332690 wait complete. 2023-10-14T11:58:46.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.387+0000 7f7913ee9700 1 Processor -- start 2023-10-14T11:58:46.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.387+0000 7f7913ee9700 1 -- start start 2023-10-14T11:58:46.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.388+0000 7f7913ee9700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f790c06ab40 0x7f790c1e1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:46.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.388+0000 7f7913ee9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f790c06f1c0 0x7f790c1de6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:46.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.388+0000 7f7913ee9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f790c158a30 0x7f790c136f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:46.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.388+0000 7f7913ee9700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f790c1df190 con 0x7f790c158a30 2023-10-14T11:58:46.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.388+0000 7f7913ee9700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f790c1374d0 con 0x7f790c06f1c0 2023-10-14T11:58:46.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.388+0000 7f7913ee9700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f790c1dded0 con 0x7f790c06ab40 2023-10-14T11:58:46.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.388+0000 7f7912486700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f790c158a30 0x7f790c136f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:46.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.388+0000 7f7912486700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f790c158a30 0x7f790c136f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45730/0 (socket says 172.21.15.78:45730) 2023-10-14T11:58:46.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.388+0000 7f7912486700 1 -- 172.21.15.78:0/1059044789 learned_addr learned my addr 172.21.15.78:0/1059044789 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:46.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.388+0000 7f7911484700 1 --2- 172.21.15.78:0/1059044789 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f790c06f1c0 0x7f790c1de6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:46.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.388+0000 7f7911c85700 1 --2- 172.21.15.78:0/1059044789 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f790c06ab40 0x7f790c1e1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:46.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.388+0000 7f7912486700 1 -- 172.21.15.78:0/1059044789 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f790c06ab40 msgr2=0x7f790c1e1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:46.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.388+0000 7f7912486700 1 --2- 172.21.15.78:0/1059044789 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f790c06ab40 0x7f790c1e1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:46.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.388+0000 7f7912486700 1 -- 172.21.15.78:0/1059044789 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f790c06f1c0 msgr2=0x7f790c1de6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:46.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.388+0000 7f7912486700 1 --2- 172.21.15.78:0/1059044789 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f790c06f1c0 0x7f790c1de6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:46.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.388+0000 7f7912486700 1 -- 172.21.15.78:0/1059044789 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7904009a70 con 0x7f790c158a30 2023-10-14T11:58:46.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.389+0000 7f7912486700 1 --2- 172.21.15.78:0/1059044789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f790c158a30 0x7f790c136f90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f790400b730 tx=0x7f7904004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:46.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.389+0000 7f79027fc700 1 -- 172.21.15.78:0/1059044789 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7904012630 con 0x7f790c158a30 2023-10-14T11:58:46.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.389+0000 7f79027fc700 1 -- 172.21.15.78:0/1059044789 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7904012790 con 0x7f790c158a30 2023-10-14T11:58:46.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.389+0000 7f7913ee9700 1 -- 172.21.15.78:0/1059044789 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f790c1edea0 con 0x7f790c158a30 2023-10-14T11:58:46.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.389+0000 7f79027fc700 1 -- 172.21.15.78:0/1059044789 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7904012b40 con 0x7f790c158a30 2023-10-14T11:58:46.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.389+0000 7f7913ee9700 1 -- 172.21.15.78:0/1059044789 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f790c1551f0 con 0x7f790c158a30 2023-10-14T11:58:46.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.390+0000 7f7913ee9700 1 -- 172.21.15.78:0/1059044789 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f790c074b80 con 0x7f790c158a30 2023-10-14T11:58:46.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.390+0000 7f79027fc700 1 -- 172.21.15.78:0/1059044789 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7904012ca0 con 0x7f790c158a30 2023-10-14T11:58:46.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.390+0000 7f79027fc700 1 --2- 172.21.15.78:0/1059044789 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f78f006af00 0x7f78f006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:46.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.391+0000 7f79027fc700 1 -- 172.21.15.78:0/1059044789 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f790408b8a0 con 0x7f790c158a30 2023-10-14T11:58:46.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.391+0000 7f7911c85700 1 --2- 172.21.15.78:0/1059044789 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f78f006af00 0x7f78f006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:46.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.391+0000 7f7911c85700 1 --2- 172.21.15.78:0/1059044789 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f78f006af00 0x7f78f006d3c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f78fc000f60 tx=0x7f78fc007c40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:46.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.395+0000 7f79027fc700 1 -- 172.21.15.78:0/1059044789 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f79040569e0 con 0x7f790c158a30 2023-10-14T11:58:46.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.584+0000 7f7913ee9700 1 -- 172.21.15.78:0/1059044789 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f790c154820 con 0x7f790c158a30 2023-10-14T11:58:46.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.584+0000 7f79027fc700 1 -- 172.21.15.78:0/1059044789 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f790405a000 con 0x7f790c158a30 2023-10-14T11:58:46.587 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2023-10-14T11:58:46.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.586+0000 7f7913ee9700 1 -- 172.21.15.78:0/1059044789 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f78f006af00 msgr2=0x7f78f006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:46.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.586+0000 7f7913ee9700 1 --2- 172.21.15.78:0/1059044789 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f78f006af00 0x7f78f006d3c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f78fc000f60 tx=0x7f78fc007c40 comp rx=0 tx=0).stop 2023-10-14T11:58:46.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.586+0000 7f7913ee9700 1 -- 172.21.15.78:0/1059044789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f790c158a30 msgr2=0x7f790c136f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:46.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.586+0000 7f7913ee9700 1 --2- 172.21.15.78:0/1059044789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f790c158a30 0x7f790c136f90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f790400b730 tx=0x7f7904004ad0 comp rx=0 tx=0).stop 2023-10-14T11:58:46.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.586+0000 7f7913ee9700 1 -- 172.21.15.78:0/1059044789 shutdown_connections 2023-10-14T11:58:46.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.586+0000 7f7913ee9700 1 --2- 172.21.15.78:0/1059044789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f790c158a30 0x7f790c136f90 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:46.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.586+0000 7f7913ee9700 1 --2- 172.21.15.78:0/1059044789 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f790c06ab40 0x7f790c1e1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:46.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.586+0000 7f7913ee9700 1 --2- 172.21.15.78:0/1059044789 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f78f006af00 0x7f78f006d3c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:46.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.586+0000 7f7913ee9700 1 --2- 172.21.15.78:0/1059044789 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f790c06f1c0 0x7f790c1de6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:46.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.586+0000 7f7913ee9700 1 -- 172.21.15.78:0/1059044789 >> 172.21.15.78:0/1059044789 conn(0x7f790c0c39c0 msgr2=0x7f790c0c23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:46.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.586+0000 7f7913ee9700 1 -- 172.21.15.78:0/1059044789 shutdown_connections 2023-10-14T11:58:46.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.586+0000 7f7913ee9700 1 -- 172.21.15.78:0/1059044789 wait complete. 2023-10-14T11:58:46.601 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-14T11:58:46.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.720+0000 7f14ca183700 1 Processor -- start 2023-10-14T11:58:46.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.720+0000 7f14ca183700 1 -- start start 2023-10-14T11:58:46.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.720+0000 7f14ca183700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f14c413dbe0 0x7f14c406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:46.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.720+0000 7f14ca183700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f14c40c4540 0x7f14c40c02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:46.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.720+0000 7f14ca183700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f14c406ae10 0x7f14c40c0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:46.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.720+0000 7f14ca183700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14c406d330 con 0x7f14c413dbe0 2023-10-14T11:58:46.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.720+0000 7f14ca183700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14c406ebe0 con 0x7f14c406ae10 2023-10-14T11:58:46.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.720+0000 7f14ca183700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14c40c0d60 con 0x7f14c40c4540 2023-10-14T11:58:46.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.720+0000 7f14c37fe700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f14c413dbe0 0x7f14c406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:46.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.720+0000 7f14c37fe700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f14c413dbe0 0x7f14c406a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45734/0 (socket says 172.21.15.78:45734) 2023-10-14T11:58:46.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.721+0000 7f14c37fe700 1 -- 172.21.15.78:0/3509829486 learned_addr learned my addr 172.21.15.78:0/3509829486 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:46.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.721+0000 7f14c2ffd700 1 --2- 172.21.15.78:0/3509829486 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f14c40c4540 0x7f14c40c02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:46.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.721+0000 7f14c3fff700 1 --2- 172.21.15.78:0/3509829486 >> v2:172.21.15.195:3300/0 conn(0x7f14c406ae10 0x7f14c40c0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:46.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.721+0000 7f14c37fe700 1 -- 172.21.15.78:0/3509829486 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f14c40c4540 msgr2=0x7f14c40c02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:46.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.721+0000 7f14c37fe700 1 --2- 172.21.15.78:0/3509829486 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f14c40c4540 0x7f14c40c02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:46.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.721+0000 7f14c37fe700 1 -- 172.21.15.78:0/3509829486 >> v2:172.21.15.195:3300/0 conn(0x7f14c406ae10 msgr2=0x7f14c40c0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:46.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.721+0000 7f14c37fe700 1 --2- 172.21.15.78:0/3509829486 >> v2:172.21.15.195:3300/0 conn(0x7f14c406ae10 0x7f14c40c0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:46.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.721+0000 7f14c37fe700 1 -- 172.21.15.78:0/3509829486 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c41558c0 con 0x7f14c413dbe0 2023-10-14T11:58:46.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.721+0000 7f14c37fe700 1 --2- 172.21.15.78:0/3509829486 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14c413dbe0 0x7f14c406a8d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f14c406e270 tx=0x7f14b4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73aecc0162ed7bd7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:46.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.721+0000 7f14c27fc700 1 -- 172.21.15.78:0/3509829486 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b400b870 con 0x7f14c413dbe0 2023-10-14T11:58:46.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.721+0000 7f14c27fc700 1 -- 172.21.15.78:0/3509829486 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b400b9d0 con 0x7f14c413dbe0 2023-10-14T11:58:46.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.721+0000 7f14c27fc700 1 -- 172.21.15.78:0/3509829486 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b400bd60 con 0x7f14c413dbe0 2023-10-14T11:58:46.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.722+0000 7f14ca183700 1 -- 172.21.15.78:0/3509829486 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14c413dbe0 msgr2=0x7f14c406a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:46.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.722+0000 7f14ca183700 1 --2- 172.21.15.78:0/3509829486 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14c413dbe0 0x7f14c406a8d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f14c406e270 tx=0x7f14b4009bd0 comp rx=0 tx=0).stop 2023-10-14T11:58:46.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.722+0000 7f14ca183700 1 -- 172.21.15.78:0/3509829486 shutdown_connections 2023-10-14T11:58:46.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.722+0000 7f14ca183700 1 --2- 172.21.15.78:0/3509829486 >> v2:172.21.15.195:3300/0 conn(0x7f14c406ae10 0x7f14c40c0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:46.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.722+0000 7f14ca183700 1 --2- 172.21.15.78:0/3509829486 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14c413dbe0 0x7f14c406a8d0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:46.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.722+0000 7f14ca183700 1 --2- 172.21.15.78:0/3509829486 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f14c40c4540 0x7f14c40c02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:46.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.722+0000 7f14ca183700 1 -- 172.21.15.78:0/3509829486 >> 172.21.15.78:0/3509829486 conn(0x7f14c40c3a50 msgr2=0x7f14c40c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:46.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.722+0000 7f14ca183700 1 -- 172.21.15.78:0/3509829486 shutdown_connections 2023-10-14T11:58:46.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.722+0000 7f14ca183700 1 -- 172.21.15.78:0/3509829486 wait complete. 2023-10-14T11:58:46.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.723+0000 7f14ca183700 1 Processor -- start 2023-10-14T11:58:46.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.723+0000 7f14ca183700 1 -- start start 2023-10-14T11:58:46.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.723+0000 7f14ca183700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f14c406ae10 0x7f14c413bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:46.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.724+0000 7f14ca183700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f14c40c4540 0x7f14c4155eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:46.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.724+0000 7f14ca183700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14c413dbe0 0x7f14c41e95f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:46.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.724+0000 7f14ca183700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c41ecb60 con 0x7f14c413dbe0 2023-10-14T11:58:46.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.724+0000 7f14ca183700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c41ed3c0 con 0x7f14c40c4540 2023-10-14T11:58:46.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.724+0000 7f14ca183700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14c40ac470 con 0x7f14c406ae10 2023-10-14T11:58:46.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.724+0000 7f14c3fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14c413dbe0 0x7f14c41e95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:46.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.724+0000 7f14c3fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14c413dbe0 0x7f14c41e95f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45744/0 (socket says 172.21.15.78:45744) 2023-10-14T11:58:46.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.724+0000 7f14c3fff700 1 -- 172.21.15.78:0/707425376 learned_addr learned my addr 172.21.15.78:0/707425376 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:46.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.724+0000 7f14c37fe700 1 --2- 172.21.15.78:0/707425376 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f14c406ae10 0x7f14c413bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:46.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.724+0000 7f14c2ffd700 1 --2- 172.21.15.78:0/707425376 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f14c40c4540 0x7f14c4155eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:46.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.724+0000 7f14c3fff700 1 -- 172.21.15.78:0/707425376 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f14c406ae10 msgr2=0x7f14c413bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:46.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.724+0000 7f14c3fff700 1 --2- 172.21.15.78:0/707425376 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f14c406ae10 0x7f14c413bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:46.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.724+0000 7f14c3fff700 1 -- 172.21.15.78:0/707425376 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f14c40c4540 msgr2=0x7f14c4155eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:46.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.724+0000 7f14c3fff700 1 --2- 172.21.15.78:0/707425376 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f14c40c4540 0x7f14c4155eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:46.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.724+0000 7f14c3fff700 1 -- 172.21.15.78:0/707425376 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b4009550 con 0x7f14c413dbe0 2023-10-14T11:58:46.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.724+0000 7f14c3fff700 1 --2- 172.21.15.78:0/707425376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14c413dbe0 0x7f14c41e95f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f14bc007cf0 tx=0x7f14bc00e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:46.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.724+0000 7f14bbfff700 1 -- 172.21.15.78:0/707425376 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14bc00eb70 con 0x7f14c413dbe0 2023-10-14T11:58:46.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.724+0000 7f14bbfff700 1 -- 172.21.15.78:0/707425376 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14bc00ecf0 con 0x7f14c413dbe0 2023-10-14T11:58:46.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.724+0000 7f14ca183700 1 -- 172.21.15.78:0/707425376 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14c41567b0 con 0x7f14c413dbe0 2023-10-14T11:58:46.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.724+0000 7f14bbfff700 1 -- 172.21.15.78:0/707425376 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14bc014660 con 0x7f14c413dbe0 2023-10-14T11:58:46.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.724+0000 7f14ca183700 1 -- 172.21.15.78:0/707425376 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14c41ed050 con 0x7f14c413dbe0 2023-10-14T11:58:46.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.725+0000 7f14ca183700 1 -- 172.21.15.78:0/707425376 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14c41ed530 con 0x7f14c413dbe0 2023-10-14T11:58:46.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.726+0000 7f14bbfff700 1 -- 172.21.15.78:0/707425376 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f14bc0147c0 con 0x7f14c413dbe0 2023-10-14T11:58:46.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.726+0000 7f14bbfff700 1 --2- 172.21.15.78:0/707425376 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f14a406af00 0x7f14a406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:46.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.726+0000 7f14bbfff700 1 -- 172.21.15.78:0/707425376 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f14bc089b40 con 0x7f14c413dbe0 2023-10-14T11:58:46.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.726+0000 7f14c37fe700 1 --2- 172.21.15.78:0/707425376 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f14a406af00 0x7f14a406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:46.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.727+0000 7f14c37fe700 1 --2- 172.21.15.78:0/707425376 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f14a406af00 0x7f14a406d3c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f14b4000c00 tx=0x7f14b4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:46.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.731+0000 7f14bbfff700 1 -- 172.21.15.78:0/707425376 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f14bc054c80 con 0x7f14c413dbe0 2023-10-14T11:58:46.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.934+0000 7f14ca183700 1 -- 172.21.15.78:0/707425376 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f14c4074b80 con 0x7f14c413dbe0 2023-10-14T11:58:46.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.934+0000 7f14bbfff700 1 -- 172.21.15.78:0/707425376 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7f14bc00ee60 con 0x7f14c413dbe0 2023-10-14T11:58:46.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.937+0000 7f14ca183700 1 -- 172.21.15.78:0/707425376 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f14a406af00 msgr2=0x7f14a406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:46.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.937+0000 7f14ca183700 1 --2- 172.21.15.78:0/707425376 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f14a406af00 0x7f14a406d3c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f14b4000c00 tx=0x7f14b4008030 comp rx=0 tx=0).stop 2023-10-14T11:58:46.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.937+0000 7f14ca183700 1 -- 172.21.15.78:0/707425376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14c413dbe0 msgr2=0x7f14c41e95f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:46.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.937+0000 7f14ca183700 1 --2- 172.21.15.78:0/707425376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14c413dbe0 0x7f14c41e95f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f14bc007cf0 tx=0x7f14bc00e0e0 comp rx=0 tx=0).stop 2023-10-14T11:58:46.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.937+0000 7f14ca183700 1 -- 172.21.15.78:0/707425376 shutdown_connections 2023-10-14T11:58:46.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.937+0000 7f14ca183700 1 --2- 172.21.15.78:0/707425376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14c413dbe0 0x7f14c41e95f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:46.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.937+0000 7f14ca183700 1 --2- 172.21.15.78:0/707425376 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f14c406ae10 0x7f14c413bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:46.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.937+0000 7f14ca183700 1 --2- 172.21.15.78:0/707425376 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f14a406af00 0x7f14a406d3c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:46.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.937+0000 7f14ca183700 1 --2- 172.21.15.78:0/707425376 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f14c40c4540 0x7f14c4155eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:46.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.937+0000 7f14ca183700 1 -- 172.21.15.78:0/707425376 >> 172.21.15.78:0/707425376 conn(0x7f14c40c3a50 msgr2=0x7f14c40c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:46.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.937+0000 7f14ca183700 1 -- 172.21.15.78:0/707425376 shutdown_connections 2023-10-14T11:58:46.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:46.937+0000 7f14ca183700 1 -- 172.21.15.78:0/707425376 wait complete. 2023-10-14T11:58:46.942 INFO:teuthology.orchestra.run.smithi078.stderr:listed 0 entries 2023-10-14T11:58:46.951 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2023-10-14T11:58:46.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:46.951 DEBUG:teuthology.orchestra.run.smithi195:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2023-10-14T11:58:46.997 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi195.front.sepia.ceph.com 2023-10-14T11:58:46.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:58:46.997 DEBUG:teuthology.orchestra.run.smithi195:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-10-14T11:58:47.067 INFO:tasks.cephfs.cephfs_test_case:[, ] 2023-10-14T11:58:47.068 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T11:58:47.193 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.191+0000 7ffa17237700 1 Processor -- start 2023-10-14T11:58:47.193 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.191+0000 7ffa17237700 1 -- start start 2023-10-14T11:58:47.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.192+0000 7ffa17237700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ffa1006f7b0 0x7ffa10072360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:47.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.192+0000 7ffa17237700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffa101526a0 0x7ffa10152b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:47.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.192+0000 7ffa17237700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa10155c90 con 0x7ffa100c1be0 2023-10-14T11:58:47.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.192+0000 7ffa17237700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa10155f70 con 0x7ffa1006f7b0 2023-10-14T11:58:47.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.192+0000 7ffa17237700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa10156250 con 0x7ffa101526a0 2023-10-14T11:58:47.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.192+0000 7ffa14fd3700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ffa1006f7b0 0x7ffa10072360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:47.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.192+0000 7ffa0ffff700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7ffa100c1be0 0x7ffa1014f840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50624/0 (socket says 172.21.15.78:50624) 2023-10-14T11:58:47.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.192+0000 7ffa0ffff700 1 -- 172.21.15.78:0/813927880 learned_addr learned my addr 172.21.15.78:0/813927880 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:47.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.192+0000 7ffa157d4700 1 --2- 172.21.15.78:0/813927880 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffa101526a0 0x7ffa10152b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:47.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.192+0000 7ffa14fd3700 1 -- 172.21.15.78:0/813927880 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffa101526a0 msgr2=0x7ffa10152b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:47.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.192+0000 7ffa14fd3700 1 --2- 172.21.15.78:0/813927880 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffa101526a0 0x7ffa10152b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:47.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.192+0000 7ffa14fd3700 1 -- 172.21.15.78:0/813927880 >> v1:172.21.15.78:6789/0 conn(0x7ffa100c1be0 legacy=0x7ffa1014f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:47.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.193+0000 7ffa14fd3700 1 -- 172.21.15.78:0/813927880 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa10156530 con 0x7ffa1006f7b0 2023-10-14T11:58:47.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.193+0000 7ffa14fd3700 1 --2- 172.21.15.78:0/813927880 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa1006f7b0 0x7ffa10072360 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ffa04009840 tx=0x7ffa04009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28497ab8bf0a8f74 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:47.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.193+0000 7ffa0f7fe700 1 -- 172.21.15.78:0/813927880 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa04004230 con 0x7ffa1006f7b0 2023-10-14T11:58:47.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.193+0000 7ffa0f7fe700 1 -- 172.21.15.78:0/813927880 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa04004390 con 0x7ffa1006f7b0 2023-10-14T11:58:47.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.193+0000 7ffa0f7fe700 1 -- 172.21.15.78:0/813927880 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa040036a0 con 0x7ffa1006f7b0 2023-10-14T11:58:47.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.193+0000 7ffa17237700 1 -- 172.21.15.78:0/813927880 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa1006f7b0 msgr2=0x7ffa10072360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:47.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.193+0000 7ffa17237700 1 --2- 172.21.15.78:0/813927880 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa1006f7b0 0x7ffa10072360 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ffa04009840 tx=0x7ffa04009b50 comp rx=0 tx=0).stop 2023-10-14T11:58:47.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.193+0000 7ffa17237700 1 -- 172.21.15.78:0/813927880 shutdown_connections 2023-10-14T11:58:47.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.193+0000 7ffa17237700 1 --2- 172.21.15.78:0/813927880 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffa101526a0 0x7ffa10152b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:47.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.193+0000 7ffa17237700 1 --2- 172.21.15.78:0/813927880 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa1006f7b0 0x7ffa10072360 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:47.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.193+0000 7ffa17237700 1 -- 172.21.15.78:0/813927880 >> 172.21.15.78:0/813927880 conn(0x7ffa100c3ac0 msgr2=0x7ffa100c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:47.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.193+0000 7ffa17237700 1 -- 172.21.15.78:0/813927880 shutdown_connections 2023-10-14T11:58:47.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.193+0000 7ffa17237700 1 -- 172.21.15.78:0/813927880 wait complete. 2023-10-14T11:58:47.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.195+0000 7ffa17237700 1 Processor -- start 2023-10-14T11:58:47.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.195+0000 7ffa17237700 1 -- start start 2023-10-14T11:58:47.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.195+0000 7ffa17237700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa1006f7b0 0x7ffa10151b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:47.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.195+0000 7ffa17237700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffa100c1be0 0x7ffa10141880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:47.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.195+0000 7ffa17237700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffa101526a0 0x7ffa10141dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:47.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.195+0000 7ffa17237700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa10142300 con 0x7ffa1006f7b0 2023-10-14T11:58:47.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.195+0000 7ffa17237700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa101e9dc0 con 0x7ffa100c1be0 2023-10-14T11:58:47.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.195+0000 7ffa17237700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa101ea060 con 0x7ffa101526a0 2023-10-14T11:58:47.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.196+0000 7ffa14fd3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa1006f7b0 0x7ffa10151b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:47.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.196+0000 7ffa14fd3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa1006f7b0 0x7ffa10151b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45754/0 (socket says 172.21.15.78:45754) 2023-10-14T11:58:47.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.196+0000 7ffa14fd3700 1 -- 172.21.15.78:0/976819713 learned_addr learned my addr 172.21.15.78:0/976819713 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:47.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.196+0000 7ffa157d4700 1 --2- 172.21.15.78:0/976819713 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffa101526a0 0x7ffa10141dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:47.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.196+0000 7ffa0ffff700 1 --2- 172.21.15.78:0/976819713 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffa100c1be0 0x7ffa10141880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:47.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.196+0000 7ffa14fd3700 1 -- 172.21.15.78:0/976819713 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffa101526a0 msgr2=0x7ffa10141dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:47.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.196+0000 7ffa14fd3700 1 --2- 172.21.15.78:0/976819713 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffa101526a0 0x7ffa10141dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:47.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.196+0000 7ffa14fd3700 1 -- 172.21.15.78:0/976819713 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffa100c1be0 msgr2=0x7ffa10141880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:47.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.196+0000 7ffa14fd3700 1 --2- 172.21.15.78:0/976819713 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffa100c1be0 0x7ffa10141880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:47.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.196+0000 7ffa14fd3700 1 -- 172.21.15.78:0/976819713 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa040094a0 con 0x7ffa1006f7b0 2023-10-14T11:58:47.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.196+0000 7ffa157d4700 1 --2- 172.21.15.78:0/976819713 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffa101526a0 0x7ffa10141dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:47.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.196+0000 7ffa0ffff700 1 --2- 172.21.15.78:0/976819713 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffa100c1be0 0x7ffa10141880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:47.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.196+0000 7ffa14fd3700 1 --2- 172.21.15.78:0/976819713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa1006f7b0 0x7ffa10151b50 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ffa040090b0 tx=0x7ffa04004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:47.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.196+0000 7ffa0d7fa700 1 -- 172.21.15.78:0/976819713 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa04011830 con 0x7ffa1006f7b0 2023-10-14T11:58:47.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.196+0000 7ffa0d7fa700 1 -- 172.21.15.78:0/976819713 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa0401f070 con 0x7ffa1006f7b0 2023-10-14T11:58:47.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.196+0000 7ffa17237700 1 -- 172.21.15.78:0/976819713 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa101ea340 con 0x7ffa1006f7b0 2023-10-14T11:58:47.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.196+0000 7ffa0d7fa700 1 -- 172.21.15.78:0/976819713 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa04011bd0 con 0x7ffa1006f7b0 2023-10-14T11:58:47.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.196+0000 7ffa17237700 1 -- 172.21.15.78:0/976819713 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa101ea900 con 0x7ffa1006f7b0 2023-10-14T11:58:47.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.197+0000 7ffa17237700 1 -- 172.21.15.78:0/976819713 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa10074b80 con 0x7ffa1006f7b0 2023-10-14T11:58:47.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.198+0000 7ffa0d7fa700 1 -- 172.21.15.78:0/976819713 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ffa04011d30 con 0x7ffa1006f7b0 2023-10-14T11:58:47.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.198+0000 7ffa0d7fa700 1 --2- 172.21.15.78:0/976819713 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff9f406af30 0x7ff9f406d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:47.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.198+0000 7ffa0ffff700 1 --2- 172.21.15.78:0/976819713 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff9f406af30 0x7ff9f406d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:47.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.198+0000 7ffa0d7fa700 1 -- 172.21.15.78:0/976819713 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffa04021070 con 0x7ffa1006f7b0 2023-10-14T11:58:47.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.198+0000 7ffa0ffff700 1 --2- 172.21.15.78:0/976819713 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff9f406af30 0x7ff9f406d3f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ffa10143a50 tx=0x7ff9fc007e30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:47.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.203+0000 7ffa0d7fa700 1 -- 172.21.15.78:0/976819713 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ffa0408e030 con 0x7ffa1006f7b0 2023-10-14T11:58:47.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.436+0000 7ffa17237700 1 -- 172.21.15.78:0/976819713 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa101495d0 con 0x7ffa1006f7b0 2023-10-14T11:58:47.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.437+0000 7ffa0d7fa700 1 -- 172.21.15.78:0/976819713 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4904 (secure 0 0 0) 0x7ffa04056050 con 0x7ffa1006f7b0 2023-10-14T11:58:47.439 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:47.439 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/353043244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":353043244},{"type":"v1","addr":"172.21.15.78:6836","nonce":353043244}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6834/3389817343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":3389817343},{"type":"v1","addr":"172.21.15.195:6834","nonce":3389817343}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4267,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/460176701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6833","nonce":460176701},{"type":"v1","addr":"172.21.15.195:6835","nonce":460176701}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T11:58:14.535101+0000","modified":"2023-10-14T11:58:17.559202+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/920300787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":920300787},{"type":"v1","addr":"172.21.15.78:6837","nonce":920300787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-14T11:58:47.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.439+0000 7ffa17237700 1 -- 172.21.15.78:0/976819713 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff9f406af30 msgr2=0x7ff9f406d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:47.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.439+0000 7ffa17237700 1 --2- 172.21.15.78:0/976819713 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff9f406af30 0x7ff9f406d3f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ffa10143a50 tx=0x7ff9fc007e30 comp rx=0 tx=0).stop 2023-10-14T11:58:47.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.439+0000 7ffa17237700 1 -- 172.21.15.78:0/976819713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa1006f7b0 msgr2=0x7ffa10151b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:47.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.439+0000 7ffa17237700 1 --2- 172.21.15.78:0/976819713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa1006f7b0 0x7ffa10151b50 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ffa040090b0 tx=0x7ffa04004750 comp rx=0 tx=0).stop 2023-10-14T11:58:47.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.439+0000 7ffa17237700 1 -- 172.21.15.78:0/976819713 shutdown_connections 2023-10-14T11:58:47.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.439+0000 7ffa17237700 1 --2- 172.21.15.78:0/976819713 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffa101526a0 0x7ffa10141dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:47.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.439+0000 7ffa17237700 1 --2- 172.21.15.78:0/976819713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa1006f7b0 0x7ffa10151b50 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:47.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.439+0000 7ffa17237700 1 --2- 172.21.15.78:0/976819713 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff9f406af30 0x7ff9f406d3f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:47.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.439+0000 7ffa17237700 1 --2- 172.21.15.78:0/976819713 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffa100c1be0 0x7ffa10141880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:47.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.439+0000 7ffa17237700 1 -- 172.21.15.78:0/976819713 >> 172.21.15.78:0/976819713 conn(0x7ffa100c3ac0 msgr2=0x7ffa10071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:47.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.439+0000 7ffa17237700 1 -- 172.21.15.78:0/976819713 shutdown_connections 2023-10-14T11:58:47.444 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.439+0000 7ffa17237700 1 -- 172.21.15.78:0/976819713 wait complete. 2023-10-14T11:58:47.444 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 8 2023-10-14T11:58:47.453 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T11:58:47.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.571+0000 7ff04590f700 1 Processor -- start 2023-10-14T11:58:47.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.571+0000 7ff04590f700 1 -- start start 2023-10-14T11:58:47.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.572+0000 7ff04590f700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff040154b90 0x7ff0400c5150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:47.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.572+0000 7ff04590f700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff04006cd00 0x7ff04006b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:47.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.572+0000 7ff04590f700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0400c9970 con 0x7ff04014b7e0 2023-10-14T11:58:47.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.572+0000 7ff04590f700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0400c4e40 con 0x7ff040154b90 2023-10-14T11:58:47.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.572+0000 7ff04590f700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0400c81c0 con 0x7ff04006cd00 2023-10-14T11:58:47.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.572+0000 7ff03e7fc700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff040154b90 0x7ff0400c5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:47.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.572+0000 7ff03effd700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7ff04014b7e0 0x7ff0400c9260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50638/0 (socket says 172.21.15.78:50638) 2023-10-14T11:58:47.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.572+0000 7ff03effd700 1 -- 172.21.15.78:0/1859950550 learned_addr learned my addr 172.21.15.78:0/1859950550 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:47.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.572+0000 7ff03e7fc700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff040154b90 0x7ff0400c5150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45764/0 (socket says 172.21.15.78:45764) 2023-10-14T11:58:47.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.572+0000 7ff03f7fe700 1 --2- 172.21.15.78:0/1859950550 >> v2:172.21.15.195:3300/0 conn(0x7ff04006cd00 0x7ff04006b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:47.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.572+0000 7ff03e7fc700 1 -- 172.21.15.78:0/1859950550 >> v2:172.21.15.195:3300/0 conn(0x7ff04006cd00 msgr2=0x7ff04006b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:47.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.572+0000 7ff03e7fc700 1 --2- 172.21.15.78:0/1859950550 >> v2:172.21.15.195:3300/0 conn(0x7ff04006cd00 0x7ff04006b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:47.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.572+0000 7ff03e7fc700 1 -- 172.21.15.78:0/1859950550 >> v1:172.21.15.78:6789/0 conn(0x7ff04014b7e0 legacy=0x7ff0400c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:47.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.572+0000 7ff03e7fc700 1 -- 172.21.15.78:0/1859950550 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0400c84a0 con 0x7ff040154b90 2023-10-14T11:58:47.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.572+0000 7ff03dffb700 1 -- 172.21.15.78:0/1859950550 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2853931493 0 0) 0x7ff0400c9970 con 0x7ff04014b7e0 2023-10-14T11:58:47.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.572+0000 7ff03e7fc700 1 --2- 172.21.15.78:0/1859950550 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff040154b90 0x7ff0400c5150 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff04006aec0 tx=0x7ff030009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=865002ed44a274d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:47.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.573+0000 7ff03dffb700 1 -- 172.21.15.78:0/1859950550 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff030011830 con 0x7ff040154b90 2023-10-14T11:58:47.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.573+0000 7ff03dffb700 1 -- 172.21.15.78:0/1859950550 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff030011990 con 0x7ff040154b90 2023-10-14T11:58:47.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.573+0000 7ff03dffb700 1 -- 172.21.15.78:0/1859950550 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff030011d20 con 0x7ff040154b90 2023-10-14T11:58:47.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.573+0000 7ff04590f700 1 -- 172.21.15.78:0/1859950550 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff040154b90 msgr2=0x7ff0400c5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:47.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.573+0000 7ff04590f700 1 --2- 172.21.15.78:0/1859950550 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff040154b90 0x7ff0400c5150 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff04006aec0 tx=0x7ff030009b20 comp rx=0 tx=0).stop 2023-10-14T11:58:47.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.573+0000 7ff04590f700 1 -- 172.21.15.78:0/1859950550 shutdown_connections 2023-10-14T11:58:47.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.573+0000 7ff04590f700 1 --2- 172.21.15.78:0/1859950550 >> v2:172.21.15.195:3300/0 conn(0x7ff04006cd00 0x7ff04006b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:47.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.573+0000 7ff04590f700 1 --2- 172.21.15.78:0/1859950550 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff040154b90 0x7ff0400c5150 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:47.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.573+0000 7ff04590f700 1 -- 172.21.15.78:0/1859950550 >> 172.21.15.78:0/1859950550 conn(0x7ff0400ca2c0 msgr2=0x7ff0400ca6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:47.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.573+0000 7ff04590f700 1 -- 172.21.15.78:0/1859950550 shutdown_connections 2023-10-14T11:58:47.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.573+0000 7ff04590f700 1 -- 172.21.15.78:0/1859950550 wait complete. 2023-10-14T11:58:47.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.574+0000 7ff04590f700 1 Processor -- start 2023-10-14T11:58:47.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.575+0000 7ff04590f700 1 -- start start 2023-10-14T11:58:47.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.575+0000 7ff04590f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff04006cd00 0x7ff0400af220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:47.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.575+0000 7ff04590f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff04014b7e0 0x7ff0400af760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:47.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.575+0000 7ff04590f700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff040154b90 0x7ff0401ed060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:47.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.575+0000 7ff04590f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0400c1030 con 0x7ff04006cd00 2023-10-14T11:58:47.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.575+0000 7ff04590f700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff040157060 con 0x7ff04014b7e0 2023-10-14T11:58:47.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.575+0000 7ff04590f700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0401456f0 con 0x7ff040154b90 2023-10-14T11:58:47.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.575+0000 7ff03effd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff04006cd00 0x7ff0400af220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:47.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.575+0000 7ff03effd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff04006cd00 0x7ff0400af220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45768/0 (socket says 172.21.15.78:45768) 2023-10-14T11:58:47.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.575+0000 7ff03effd700 1 -- 172.21.15.78:0/2192805757 learned_addr learned my addr 172.21.15.78:0/2192805757 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:47.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.575+0000 7ff03e7fc700 1 --2- 172.21.15.78:0/2192805757 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff04014b7e0 0x7ff0400af760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:47.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.575+0000 7ff03f7fe700 1 --2- 172.21.15.78:0/2192805757 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff040154b90 0x7ff0401ed060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:47.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.576+0000 7ff03effd700 1 -- 172.21.15.78:0/2192805757 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff040154b90 msgr2=0x7ff0401ed060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:47.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.576+0000 7ff03effd700 1 --2- 172.21.15.78:0/2192805757 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff040154b90 0x7ff0401ed060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:47.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.576+0000 7ff03effd700 1 -- 172.21.15.78:0/2192805757 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff04014b7e0 msgr2=0x7ff0400af760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:47.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.576+0000 7ff03effd700 1 --2- 172.21.15.78:0/2192805757 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff04014b7e0 0x7ff0400af760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:47.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.576+0000 7ff03effd700 1 -- 172.21.15.78:0/2192805757 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0300094a0 con 0x7ff04006cd00 2023-10-14T11:58:47.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.576+0000 7ff03e7fc700 1 --2- 172.21.15.78:0/2192805757 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff04014b7e0 0x7ff0400af760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:47.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.576+0000 7ff03effd700 1 --2- 172.21.15.78:0/2192805757 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff04006cd00 0x7ff0400af220 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff0340011e0 tx=0x7ff03400cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:47.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.576+0000 7ff027fff700 1 -- 172.21.15.78:0/2192805757 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff034004c80 con 0x7ff04006cd00 2023-10-14T11:58:47.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.576+0000 7ff027fff700 1 -- 172.21.15.78:0/2192805757 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff034004e00 con 0x7ff04006cd00 2023-10-14T11:58:47.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.576+0000 7ff04590f700 1 -- 172.21.15.78:0/2192805757 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0401462b0 con 0x7ff04006cd00 2023-10-14T11:58:47.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.576+0000 7ff027fff700 1 -- 172.21.15.78:0/2192805757 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff034007310 con 0x7ff04006cd00 2023-10-14T11:58:47.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.576+0000 7ff04590f700 1 -- 172.21.15.78:0/2192805757 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0401467e0 con 0x7ff04006cd00 2023-10-14T11:58:47.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.577+0000 7ff04590f700 1 -- 172.21.15.78:0/2192805757 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0400a9a80 con 0x7ff04006cd00 2023-10-14T11:58:47.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.577+0000 7ff027fff700 1 -- 172.21.15.78:0/2192805757 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff034007470 con 0x7ff04006cd00 2023-10-14T11:58:47.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.578+0000 7ff027fff700 1 --2- 172.21.15.78:0/2192805757 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff02c06af00 0x7ff02c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:47.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.578+0000 7ff027fff700 1 -- 172.21.15.78:0/2192805757 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff034089320 con 0x7ff04006cd00 2023-10-14T11:58:47.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.578+0000 7ff03e7fc700 1 --2- 172.21.15.78:0/2192805757 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff02c06af00 0x7ff02c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:47.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.578+0000 7ff03e7fc700 1 --2- 172.21.15.78:0/2192805757 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff02c06af00 0x7ff02c06d3c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff0300090b0 tx=0x7ff030008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:47.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.582+0000 7ff027fff700 1 -- 172.21.15.78:0/2192805757 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff034054460 con 0x7ff04006cd00 2023-10-14T11:58:47.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.773+0000 7ff04590f700 1 -- 172.21.15.78:0/2192805757 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff040074b80 con 0x7ff04006cd00 2023-10-14T11:58:47.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.774+0000 7ff027fff700 1 -- 172.21.15.78:0/2192805757 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16481 (secure 0 0 0) 0x7ff034057a80 con 0x7ff04006cd00 2023-10-14T11:58:47.777 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:47.777 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":21,"fsid":"4c5dc134-3092-406a-a1e1-13e6d9e926e8","created":"2023-10-14T11:57:57.363750+0000","modified":"2023-10-14T11:58:16.139413+0000","last_up_change":"2023-10-14T11:58:06.982472+0000","last_in_change":"2023-10-14T11:57:59.821373+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T11:58:09.001200+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T11:58:12.778573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T11:58:13.490286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"625a5fc0-9474-4eae-b80e-14c27a74ad19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6801","nonce":3884969780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6803","nonce":3884969780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6807","nonce":3884969780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6805","nonce":3884969780}]},"public_addr":"172.21.15.78:6801/3884969780","cluster_addr":"172.21.15.78:6803/3884969780","heartbeat_back_addr":"172.21.15.78:6807/3884969780","heartbeat_front_addr":"172.21.15.78:6805/3884969780","state":["exists","up"]},{"osd":1,"uuid":"49d63db1-2981-49ab-9c40-c50fc744b982","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6809","nonce":2176928406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6811","nonce":2176928406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6815","nonce":2176928406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6813","nonce":2176928406}]},"public_addr":"172.21.15.78:6809/2176928406","cluster_addr":"172.21.15.78:6811/2176928406","heartbeat_back_addr":"172.21.15.78:6815/2176928406","heartbeat_front_addr":"172.21.15.78:6813/2176928406","state":["exists","up"]},{"osd":2,"uuid":"1d7d0b29-cbd1-4717-9fe6-765a8d4998b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6825","nonce":3647044577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6827","nonce":3647044577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6831","nonce":3647044577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6829","nonce":3647044577}]},"public_addr":"172.21.15.78:6825/3647044577","cluster_addr":"172.21.15.78:6827/3647044577","heartbeat_back_addr":"172.21.15.78:6831/3647044577","heartbeat_front_addr":"172.21.15.78:6829/3647044577","state":["exists","up"]},{"osd":3,"uuid":"2f75ded1-e9e2-40e7-99ae-45e1e425e1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6817","nonce":3584112288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6819","nonce":3584112288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6823","nonce":3584112288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6821","nonce":3584112288}]},"public_addr":"172.21.15.78:6817/3584112288","cluster_addr":"172.21.15.78:6819/3584112288","heartbeat_back_addr":"172.21.15.78:6823/3584112288","heartbeat_front_addr":"172.21.15.78:6821/3584112288","state":["exists","up"]},{"osd":4,"uuid":"5130b432-8358-46cb-8a31-3064c976f6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6801","nonce":3732756301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6803","nonce":3732756301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6807","nonce":3732756301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6805","nonce":3732756301}]},"public_addr":"172.21.15.195:6801/3732756301","cluster_addr":"172.21.15.195:6803/3732756301","heartbeat_back_addr":"172.21.15.195:6807/3732756301","heartbeat_front_addr":"172.21.15.195:6805/3732756301","state":["exists","up"]},{"osd":5,"uuid":"60e234a9-a506-4e40-9c19-49a58f3cf28c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6825","nonce":2857706411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6827","nonce":2857706411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6831","nonce":2857706411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6829","nonce":2857706411}]},"public_addr":"172.21.15.195:6825/2857706411","cluster_addr":"172.21.15.195:6827/2857706411","heartbeat_back_addr":"172.21.15.195:6831/2857706411","heartbeat_front_addr":"172.21.15.195:6829/2857706411","state":["exists","up"]},{"osd":6,"uuid":"df7d2dd6-26b6-4eb1-96e5-94d1f3955b33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6810","nonce":72554374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6813","nonce":72554374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6820","nonce":72554374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6817","nonce":72554374}]},"public_addr":"172.21.15.195:6810/72554374","cluster_addr":"172.21.15.195:6813/72554374","heartbeat_back_addr":"172.21.15.195:6820/72554374","heartbeat_front_addr":"172.21.15.195:6817/72554374","state":["exists","up"]},{"osd":7,"uuid":"47a9d4de-f56d-4aa4-9900-c4a9cf9b2f97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6809","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6811","nonce":482472979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6816","nonce":482472979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6823","nonce":482472979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6821","nonce":482472979}]},"public_addr":"172.21.15.195:6811/482472979","cluster_addr":"172.21.15.195:6816/482472979","heartbeat_back_addr":"172.21.15.195:6823/482472979","heartbeat_front_addr":"172.21.15.195:6821/482472979","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.754003+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.005994+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.043603+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.024506+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.062585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.033767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.994825+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.038300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T11:58:47.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.776+0000 7ff04590f700 1 -- 172.21.15.78:0/2192805757 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff02c06af00 msgr2=0x7ff02c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:47.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.776+0000 7ff04590f700 1 --2- 172.21.15.78:0/2192805757 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff02c06af00 0x7ff02c06d3c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff0300090b0 tx=0x7ff030008030 comp rx=0 tx=0).stop 2023-10-14T11:58:47.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.776+0000 7ff04590f700 1 -- 172.21.15.78:0/2192805757 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff04006cd00 msgr2=0x7ff0400af220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:47.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.776+0000 7ff04590f700 1 --2- 172.21.15.78:0/2192805757 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff04006cd00 0x7ff0400af220 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff0340011e0 tx=0x7ff03400cae0 comp rx=0 tx=0).stop 2023-10-14T11:58:47.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.776+0000 7ff04590f700 1 -- 172.21.15.78:0/2192805757 shutdown_connections 2023-10-14T11:58:47.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.776+0000 7ff04590f700 1 --2- 172.21.15.78:0/2192805757 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff040154b90 0x7ff0401ed060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:47.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.776+0000 7ff04590f700 1 --2- 172.21.15.78:0/2192805757 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff04006cd00 0x7ff0400af220 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:47.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.776+0000 7ff04590f700 1 --2- 172.21.15.78:0/2192805757 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff02c06af00 0x7ff02c06d3c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:47.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.777+0000 7ff04590f700 1 --2- 172.21.15.78:0/2192805757 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff04014b7e0 0x7ff0400af760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:47.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.777+0000 7ff04590f700 1 -- 172.21.15.78:0/2192805757 >> 172.21.15.78:0/2192805757 conn(0x7ff0400ca2c0 msgr2=0x7ff0400c59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:47.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.777+0000 7ff04590f700 1 -- 172.21.15.78:0/2192805757 shutdown_connections 2023-10-14T11:58:47.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.777+0000 7ff04590f700 1 -- 172.21.15.78:0/2192805757 wait complete. 2023-10-14T11:58:47.791 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T11:58:09.001200+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T11:58:12.778573+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T11:58:13.490286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T11:58:47.791 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T11:58:47.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.912+0000 7fc73bad1700 1 Processor -- start 2023-10-14T11:58:47.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.912+0000 7fc73bad1700 1 -- start start 2023-10-14T11:58:47.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.913+0000 7fc73bad1700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc7340a70c0 0x7fc734154960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:47.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.913+0000 7fc73bad1700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7341537b0 con 0x7fc734154ea0 2023-10-14T11:58:47.919 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.913+0000 7fc73bad1700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7340c7c50 con 0x7fc7341533d0 2023-10-14T11:58:47.919 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.913+0000 7fc73bad1700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7340c6fb0 con 0x7fc7340a70c0 2023-10-14T11:58:47.919 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.913+0000 7fc73a06e700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc734154ea0 0x7fc73415d130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50656/0 (socket says 172.21.15.78:50656) 2023-10-14T11:58:47.919 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.913+0000 7fc73a06e700 1 -- 172.21.15.78:0/2544579800 learned_addr learned my addr 172.21.15.78:0/2544579800 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:47.919 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.913+0000 7fc73906c700 1 --2- 172.21.15.78:0/2544579800 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc7340a70c0 0x7fc734154960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:47.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.913+0000 7fc73886b700 1 -- 172.21.15.78:0/2544579800 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2655774996 0 0) 0x7fc7341537b0 con 0x7fc734154ea0 2023-10-14T11:58:47.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.913+0000 7fc73886b700 1 -- 172.21.15.78:0/2544579800 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7200035b0 con 0x7fc734154ea0 2023-10-14T11:58:47.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.914+0000 7fc73886b700 1 -- 172.21.15.78:0/2544579800 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3455674567 0 0) 0x7fc7200035b0 con 0x7fc734154ea0 2023-10-14T11:58:47.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.914+0000 7fc73886b700 1 -- 172.21.15.78:0/2544579800 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc7340a70c0 msgr2=0x7fc734154960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:47.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.914+0000 7fc73886b700 1 --2- 172.21.15.78:0/2544579800 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc7340a70c0 0x7fc734154960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:47.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.914+0000 7fc73886b700 1 -- 172.21.15.78:0/2544579800 >> v1:172.21.15.195:6789/0 conn(0x7fc7341533d0 legacy=0x7fc7340c8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:47.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.914+0000 7fc73886b700 1 -- 172.21.15.78:0/2544579800 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc734153e80 con 0x7fc734154ea0 2023-10-14T11:58:47.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.914+0000 7fc73886b700 1 -- 172.21.15.78:0/2544579800 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441799917 0 0) 0x7fc72c0019f0 con 0x7fc734154ea0 2023-10-14T11:58:47.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.914+0000 7fc73886b700 1 -- 172.21.15.78:0/2544579800 >> v1:172.21.15.78:6789/0 conn(0x7fc734154ea0 legacy=0x7fc73415d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:47.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.914+0000 7fc73886b700 1 --2- 172.21.15.78:0/2544579800 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc7340a70c0 0x7fc720003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:47.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.914+0000 7fc73886b700 1 --2- 172.21.15.78:0/2544579800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc720003f70 0x7fc720004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:47.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.914+0000 7fc73886b700 1 --2- 172.21.15.78:0/2544579800 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc720004d90 0x7fc720005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:47.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.914+0000 7fc73886b700 1 -- 172.21.15.78:0/2544579800 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7340c6fb0 con 0x7fc720003f70 2023-10-14T11:58:47.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.914+0000 7fc73886b700 1 -- 172.21.15.78:0/2544579800 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72000bb20 con 0x7fc720004d90 2023-10-14T11:58:47.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.914+0000 7fc73886b700 1 -- 172.21.15.78:0/2544579800 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc72000bce0 con 0x7fc7340a70c0 2023-10-14T11:58:47.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.914+0000 7fc73906c700 1 --2- 172.21.15.78:0/2544579800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc720003f70 0x7fc720004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:47.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.914+0000 7fc73986d700 1 --2- 172.21.15.78:0/2544579800 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc7340a70c0 0x7fc720003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:47.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.914+0000 7fc73a06e700 1 --2- 172.21.15.78:0/2544579800 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc720004d90 0x7fc720005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:47.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.914+0000 7fc73906c700 1 -- 172.21.15.78:0/2544579800 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc7340a70c0 msgr2=0x7fc720003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:47.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.914+0000 7fc73906c700 1 --2- 172.21.15.78:0/2544579800 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc7340a70c0 0x7fc720003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:47.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.914+0000 7fc73906c700 1 -- 172.21.15.78:0/2544579800 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc720004d90 msgr2=0x7fc720005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:47.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.914+0000 7fc73906c700 1 --2- 172.21.15.78:0/2544579800 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc720004d90 0x7fc720005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:47.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.914+0000 7fc73906c700 1 -- 172.21.15.78:0/2544579800 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7200035b0 con 0x7fc720003f70 2023-10-14T11:58:47.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.914+0000 7fc73906c700 1 --2- 172.21.15.78:0/2544579800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc720003f70 0x7fc720004850 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc730009b60 tx=0x7fc720005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed67bd82b5fa54e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:47.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.914+0000 7fc73886b700 1 -- 172.21.15.78:0/2544579800 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc73000e6c0 con 0x7fc720003f70 2023-10-14T11:58:47.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.914+0000 7fc73886b700 1 -- 172.21.15.78:0/2544579800 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc73000e820 con 0x7fc720003f70 2023-10-14T11:58:47.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.914+0000 7fc73886b700 1 -- 172.21.15.78:0/2544579800 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc73000eb90 con 0x7fc720003f70 2023-10-14T11:58:47.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.915+0000 7fc73bad1700 1 -- 172.21.15.78:0/2544579800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc720003f70 msgr2=0x7fc720004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:47.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.915+0000 7fc73bad1700 1 --2- 172.21.15.78:0/2544579800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc720003f70 0x7fc720004850 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc730009b60 tx=0x7fc720005880 comp rx=0 tx=0).stop 2023-10-14T11:58:47.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.915+0000 7fc73a06e700 1 -- 172.21.15.78:0/2544579800 reap_dead start 2023-10-14T11:58:47.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.915+0000 7fc73bad1700 1 -- 172.21.15.78:0/2544579800 shutdown_connections 2023-10-14T11:58:47.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.915+0000 7fc73bad1700 1 -- 172.21.15.78:0/2544579800 >> 172.21.15.78:0/2544579800 conn(0x7fc7340737d0 msgr2=0x7fc734073be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:47.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.915+0000 7fc73bad1700 1 -- 172.21.15.78:0/2544579800 shutdown_connections 2023-10-14T11:58:47.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.915+0000 7fc73bad1700 1 -- 172.21.15.78:0/2544579800 wait complete. 2023-10-14T11:58:47.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.916+0000 7fc73bad1700 1 Processor -- start 2023-10-14T11:58:47.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.916+0000 7fc73bad1700 1 -- start start 2023-10-14T11:58:47.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.917+0000 7fc73bad1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc720003f70 0x7fc7341eb000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:47.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.917+0000 7fc73bad1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc7340bac30 0x7fc7340c0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:47.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.917+0000 7fc73bad1700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc7341ed100 0x7fc7341ed580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:47.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.917+0000 7fc73bad1700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7340c7f30 con 0x7fc720003f70 2023-10-14T11:58:47.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.917+0000 7fc73bad1700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7340c7a70 con 0x7fc7340bac30 2023-10-14T11:58:47.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.917+0000 7fc73bad1700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7341ecf10 con 0x7fc7341ed100 2023-10-14T11:58:47.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.917+0000 7fc73986d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc720003f70 0x7fc7341eb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:47.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.917+0000 7fc73986d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc720003f70 0x7fc7341eb000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45788/0 (socket says 172.21.15.78:45788) 2023-10-14T11:58:47.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.917+0000 7fc73986d700 1 -- 172.21.15.78:0/2306225619 learned_addr learned my addr 172.21.15.78:0/2306225619 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:47.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.917+0000 7fc73986d700 1 -- 172.21.15.78:0/2306225619 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc7341ed100 msgr2=0x7fc7341ed580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:47.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.917+0000 7fc73986d700 1 --2- 172.21.15.78:0/2306225619 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc7341ed100 0x7fc7341ed580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:47.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.917+0000 7fc73986d700 1 -- 172.21.15.78:0/2306225619 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc7340bac30 msgr2=0x7fc7340c0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:47.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.917+0000 7fc73986d700 1 --2- 172.21.15.78:0/2306225619 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc7340bac30 0x7fc7340c0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:47.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.917+0000 7fc73986d700 1 -- 172.21.15.78:0/2306225619 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7341eddf0 con 0x7fc720003f70 2023-10-14T11:58:47.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.917+0000 7fc73986d700 1 --2- 172.21.15.78:0/2306225619 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc720003f70 0x7fc7341eb000 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc72400f820 tx=0x7fc72400fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:47.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.917+0000 7fc72a7fc700 1 -- 172.21.15.78:0/2306225619 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7240052e0 con 0x7fc720003f70 2023-10-14T11:58:47.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.918+0000 7fc72a7fc700 1 -- 172.21.15.78:0/2306225619 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc724005460 con 0x7fc720003f70 2023-10-14T11:58:47.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.918+0000 7fc73bad1700 1 -- 172.21.15.78:0/2306225619 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7340bfe40 con 0x7fc720003f70 2023-10-14T11:58:47.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.918+0000 7fc72a7fc700 1 -- 172.21.15.78:0/2306225619 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7240038c0 con 0x7fc720003f70 2023-10-14T11:58:47.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.918+0000 7fc73bad1700 1 -- 172.21.15.78:0/2306225619 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7340c0400 con 0x7fc720003f70 2023-10-14T11:58:47.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.918+0000 7fc73bad1700 1 -- 172.21.15.78:0/2306225619 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc734074b80 con 0x7fc720003f70 2023-10-14T11:58:47.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.919+0000 7fc72a7fc700 1 -- 172.21.15.78:0/2306225619 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc724003a20 con 0x7fc720003f70 2023-10-14T11:58:47.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.919+0000 7fc72a7fc700 1 --2- 172.21.15.78:0/2306225619 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc71806af00 0x7fc71806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:47.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.919+0000 7fc72a7fc700 1 -- 172.21.15.78:0/2306225619 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc724089390 con 0x7fc720003f70 2023-10-14T11:58:47.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.919+0000 7fc73906c700 1 --2- 172.21.15.78:0/2306225619 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc71806af00 0x7fc71806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:47.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.920+0000 7fc73906c700 1 --2- 172.21.15.78:0/2306225619 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc71806af00 0x7fc71806d3c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc7340bfd70 tx=0x7fc730013000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:47.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:47.924+0000 7fc72a7fc700 1 -- 172.21.15.78:0/2306225619 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc7240544d0 con 0x7fc720003f70 2023-10-14T11:58:48.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.113+0000 7fc73bad1700 1 -- 172.21.15.78:0/2306225619 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc7340c5a10 con 0x7fc720003f70 2023-10-14T11:58:48.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.114+0000 7fc72a7fc700 1 -- 172.21.15.78:0/2306225619 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc724057af0 con 0x7fc720003f70 2023-10-14T11:58:48.116 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-10-14T11:58:48.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.116+0000 7fc73bad1700 1 -- 172.21.15.78:0/2306225619 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc71806af00 msgr2=0x7fc71806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:48.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.116+0000 7fc73bad1700 1 --2- 172.21.15.78:0/2306225619 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc71806af00 0x7fc71806d3c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc7340bfd70 tx=0x7fc730013000 comp rx=0 tx=0).stop 2023-10-14T11:58:48.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.116+0000 7fc73bad1700 1 -- 172.21.15.78:0/2306225619 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc720003f70 msgr2=0x7fc7341eb000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:48.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.116+0000 7fc73bad1700 1 --2- 172.21.15.78:0/2306225619 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc720003f70 0x7fc7341eb000 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc72400f820 tx=0x7fc72400fbe0 comp rx=0 tx=0).stop 2023-10-14T11:58:48.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.116+0000 7fc73bad1700 1 -- 172.21.15.78:0/2306225619 shutdown_connections 2023-10-14T11:58:48.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.116+0000 7fc73bad1700 1 --2- 172.21.15.78:0/2306225619 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc7341ed100 0x7fc7341ed580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:48.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.116+0000 7fc73bad1700 1 --2- 172.21.15.78:0/2306225619 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc720003f70 0x7fc7341eb000 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:48.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.116+0000 7fc73bad1700 1 --2- 172.21.15.78:0/2306225619 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc71806af00 0x7fc71806d3c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:48.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.116+0000 7fc73bad1700 1 --2- 172.21.15.78:0/2306225619 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc7340bac30 0x7fc7340c0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:48.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.116+0000 7fc73bad1700 1 -- 172.21.15.78:0/2306225619 >> 172.21.15.78:0/2306225619 conn(0x7fc7340737d0 msgr2=0x7fc734151d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:48.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.116+0000 7fc73bad1700 1 -- 172.21.15.78:0/2306225619 shutdown_connections 2023-10-14T11:58:48.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.116+0000 7fc73bad1700 1 -- 172.21.15.78:0/2306225619 wait complete. 2023-10-14T11:58:48.130 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T11:58:48.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.251+0000 7f9a0a7ee700 1 Processor -- start 2023-10-14T11:58:48.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.251+0000 7f9a0a7ee700 1 -- start start 2023-10-14T11:58:48.256 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.252+0000 7f9a0a7ee700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9a04152130 0x7f9a04157f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:48.256 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.252+0000 7f9a0a7ee700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9a040ab700 0x7f9a0406d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:48.257 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.252+0000 7f9a0a7ee700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9a040c2c70 0x7f9a040c6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:48.257 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.252+0000 7f9a0a7ee700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a040ab050 con 0x7f9a040c2c70 2023-10-14T11:58:48.257 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.252+0000 7f9a0a7ee700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a040abb90 con 0x7f9a04152130 2023-10-14T11:58:48.257 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.252+0000 7f9a0a7ee700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a041554d0 con 0x7f9a040ab700 2023-10-14T11:58:48.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.252+0000 7f9a08d8b700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9a040c2c70 0x7f9a040c6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:48.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.252+0000 7f9a08d8b700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9a040c2c70 0x7f9a040c6720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45800/0 (socket says 172.21.15.78:45800) 2023-10-14T11:58:48.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.252+0000 7f9a08d8b700 1 -- 172.21.15.78:0/2666763184 learned_addr learned my addr 172.21.15.78:0/2666763184 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:48.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.252+0000 7f9a037fe700 1 --2- 172.21.15.78:0/2666763184 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9a040ab700 0x7f9a0406d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:48.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.252+0000 7f9a03fff700 1 --2- 172.21.15.78:0/2666763184 >> v2:172.21.15.195:3300/0 conn(0x7f9a04152130 0x7f9a04157f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:48.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.252+0000 7f9a08d8b700 1 -- 172.21.15.78:0/2666763184 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9a040ab700 msgr2=0x7f9a0406d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:48.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.252+0000 7f9a08d8b700 1 --2- 172.21.15.78:0/2666763184 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9a040ab700 0x7f9a0406d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:48.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.252+0000 7f9a08d8b700 1 -- 172.21.15.78:0/2666763184 >> v2:172.21.15.195:3300/0 conn(0x7f9a04152130 msgr2=0x7f9a04157f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:48.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.252+0000 7f9a08d8b700 1 --2- 172.21.15.78:0/2666763184 >> v2:172.21.15.195:3300/0 conn(0x7f9a04152130 0x7f9a04157f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:48.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.252+0000 7f9a08d8b700 1 -- 172.21.15.78:0/2666763184 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a041557b0 con 0x7f9a040c2c70 2023-10-14T11:58:48.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.252+0000 7f9a03fff700 1 --2- 172.21.15.78:0/2666763184 >> v2:172.21.15.195:3300/0 conn(0x7f9a04152130 0x7f9a04157f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:48.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.252+0000 7f9a037fe700 1 --2- 172.21.15.78:0/2666763184 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9a040ab700 0x7f9a0406d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:48.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.252+0000 7f9a08d8b700 1 --2- 172.21.15.78:0/2666763184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a040c2c70 0x7f9a040c6720 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9a04152760 tx=0x7f99fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=466a712452ab6f3b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:48.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.252+0000 7f9a02ffd700 1 -- 172.21.15.78:0/2666763184 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99fc00e070 con 0x7f9a040c2c70 2023-10-14T11:58:48.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.253+0000 7f9a02ffd700 1 -- 172.21.15.78:0/2666763184 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99fc00b9c0 con 0x7f9a040c2c70 2023-10-14T11:58:48.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.253+0000 7f9a02ffd700 1 -- 172.21.15.78:0/2666763184 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99fc00bd70 con 0x7f9a040c2c70 2023-10-14T11:58:48.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.253+0000 7f9a0a7ee700 1 -- 172.21.15.78:0/2666763184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a040c2c70 msgr2=0x7f9a040c6720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:48.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.253+0000 7f9a0a7ee700 1 --2- 172.21.15.78:0/2666763184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a040c2c70 0x7f9a040c6720 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9a04152760 tx=0x7f99fc00b330 comp rx=0 tx=0).stop 2023-10-14T11:58:48.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.253+0000 7f9a0a7ee700 1 -- 172.21.15.78:0/2666763184 shutdown_connections 2023-10-14T11:58:48.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.253+0000 7f9a0a7ee700 1 --2- 172.21.15.78:0/2666763184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a040c2c70 0x7f9a040c6720 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:48.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.253+0000 7f9a0a7ee700 1 --2- 172.21.15.78:0/2666763184 >> v2:172.21.15.195:3300/0 conn(0x7f9a04152130 0x7f9a04157f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:48.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.253+0000 7f9a0a7ee700 1 --2- 172.21.15.78:0/2666763184 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9a040ab700 0x7f9a0406d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:48.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.253+0000 7f9a0a7ee700 1 -- 172.21.15.78:0/2666763184 >> 172.21.15.78:0/2666763184 conn(0x7f9a040742a0 msgr2=0x7f9a040a8490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:48.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.253+0000 7f9a0a7ee700 1 -- 172.21.15.78:0/2666763184 shutdown_connections 2023-10-14T11:58:48.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.253+0000 7f9a0a7ee700 1 -- 172.21.15.78:0/2666763184 wait complete. 2023-10-14T11:58:48.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.254+0000 7f9a0a7ee700 1 Processor -- start 2023-10-14T11:58:48.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.254+0000 7f9a0a7ee700 1 -- start start 2023-10-14T11:58:48.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.255+0000 7f9a0a7ee700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9a040c2c70 0x7f9a040c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:48.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.255+0000 7f9a0a7ee700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9a04152130 0x7f9a040c4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:48.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.255+0000 7f9a0a7ee700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a0414ad90 0x7f9a040c48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:48.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.255+0000 7f9a0a7ee700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0414e480 con 0x7f9a0414ad90 2023-10-14T11:58:48.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.255+0000 7f9a0a7ee700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0414e6d0 con 0x7f9a04152130 2023-10-14T11:58:48.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.255+0000 7f9a0a7ee700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a041e9850 con 0x7f9a040c2c70 2023-10-14T11:58:48.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.255+0000 7f9a08d8b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a0414ad90 0x7f9a040c48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:48.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.255+0000 7f9a08d8b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a0414ad90 0x7f9a040c48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45806/0 (socket says 172.21.15.78:45806) 2023-10-14T11:58:48.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.255+0000 7f9a08d8b700 1 -- 172.21.15.78:0/1186515069 learned_addr learned my addr 172.21.15.78:0/1186515069 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:48.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.255+0000 7f9a037fe700 1 --2- 172.21.15.78:0/1186515069 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9a04152130 0x7f9a040c4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:48.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.255+0000 7f9a03fff700 1 --2- 172.21.15.78:0/1186515069 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9a040c2c70 0x7f9a040c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:48.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.255+0000 7f9a08d8b700 1 -- 172.21.15.78:0/1186515069 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9a040c2c70 msgr2=0x7f9a040c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:48.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.255+0000 7f9a08d8b700 1 --2- 172.21.15.78:0/1186515069 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9a040c2c70 0x7f9a040c5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:48.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.255+0000 7f9a08d8b700 1 -- 172.21.15.78:0/1186515069 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9a04152130 msgr2=0x7f9a040c4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:48.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.255+0000 7f9a08d8b700 1 --2- 172.21.15.78:0/1186515069 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9a04152130 0x7f9a040c4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:48.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.256+0000 7f9a08d8b700 1 -- 172.21.15.78:0/1186515069 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99fc009b30 con 0x7f9a0414ad90 2023-10-14T11:58:48.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.256+0000 7f9a037fe700 1 --2- 172.21.15.78:0/1186515069 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9a04152130 0x7f9a040c4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:48.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.256+0000 7f9a03fff700 1 --2- 172.21.15.78:0/1186515069 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9a040c2c70 0x7f9a040c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:48.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.256+0000 7f9a08d8b700 1 --2- 172.21.15.78:0/1186515069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a0414ad90 0x7f9a040c48a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f99fc00b370 tx=0x7f99fc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:48.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.256+0000 7f9a00ff9700 1 -- 172.21.15.78:0/1186515069 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99fc004250 con 0x7f9a0414ad90 2023-10-14T11:58:48.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.256+0000 7f9a00ff9700 1 -- 172.21.15.78:0/1186515069 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99fc0043b0 con 0x7f9a0414ad90 2023-10-14T11:58:48.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.256+0000 7f9a0a7ee700 1 -- 172.21.15.78:0/1186515069 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a041e9b30 con 0x7f9a0414ad90 2023-10-14T11:58:48.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.256+0000 7f9a00ff9700 1 -- 172.21.15.78:0/1186515069 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99fc004760 con 0x7f9a0414ad90 2023-10-14T11:58:48.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.256+0000 7f9a0a7ee700 1 -- 172.21.15.78:0/1186515069 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a040c5cb0 con 0x7f9a0414ad90 2023-10-14T11:58:48.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.257+0000 7f9a0a7ee700 1 -- 172.21.15.78:0/1186515069 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a041ea050 con 0x7f9a0414ad90 2023-10-14T11:58:48.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.257+0000 7f9a00ff9700 1 -- 172.21.15.78:0/1186515069 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f99fc01c440 con 0x7f9a0414ad90 2023-10-14T11:58:48.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.257+0000 7f9a00ff9700 1 --2- 172.21.15.78:0/1186515069 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f99f006af00 0x7f99f006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:48.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.258+0000 7f9a00ff9700 1 -- 172.21.15.78:0/1186515069 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f99fc08ab30 con 0x7f9a0414ad90 2023-10-14T11:58:48.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.258+0000 7f9a03fff700 1 --2- 172.21.15.78:0/1186515069 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f99f006af00 0x7f99f006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:48.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.258+0000 7f9a03fff700 1 --2- 172.21.15.78:0/1186515069 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f99f006af00 0x7f99f006d3c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f99f8008910 tx=0x7f99f8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:48.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.262+0000 7f9a00ff9700 1 -- 172.21.15.78:0/1186515069 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9a041ea050 con 0x7f9a0414ad90 2023-10-14T11:58:48.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.451+0000 7f9a0a7ee700 1 -- 172.21.15.78:0/1186515069 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9a041ea050 con 0x7f9a0414ad90 2023-10-14T11:58:48.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.451+0000 7f9a00ff9700 1 -- 172.21.15.78:0/1186515069 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9a041ea050 con 0x7f9a0414ad90 2023-10-14T11:58:48.454 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-10-14T11:58:48.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.453+0000 7f9a0a7ee700 1 -- 172.21.15.78:0/1186515069 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f99f006af00 msgr2=0x7f99f006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:48.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.453+0000 7f9a0a7ee700 1 --2- 172.21.15.78:0/1186515069 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f99f006af00 0x7f99f006d3c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f99f8008910 tx=0x7f99f8008040 comp rx=0 tx=0).stop 2023-10-14T11:58:48.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.453+0000 7f9a0a7ee700 1 -- 172.21.15.78:0/1186515069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a0414ad90 msgr2=0x7f9a040c48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:48.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.453+0000 7f9a0a7ee700 1 --2- 172.21.15.78:0/1186515069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a0414ad90 0x7f9a040c48a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f99fc00b370 tx=0x7f99fc004ad0 comp rx=0 tx=0).stop 2023-10-14T11:58:48.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.453+0000 7f9a0a7ee700 1 -- 172.21.15.78:0/1186515069 shutdown_connections 2023-10-14T11:58:48.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.453+0000 7f9a0a7ee700 1 --2- 172.21.15.78:0/1186515069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a0414ad90 0x7f9a040c48a0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:48.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.453+0000 7f9a0a7ee700 1 --2- 172.21.15.78:0/1186515069 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9a040c2c70 0x7f9a040c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:48.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.453+0000 7f9a0a7ee700 1 --2- 172.21.15.78:0/1186515069 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f99f006af00 0x7f99f006d3c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:48.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.453+0000 7f9a0a7ee700 1 --2- 172.21.15.78:0/1186515069 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9a04152130 0x7f9a040c4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:48.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.453+0000 7f9a0a7ee700 1 -- 172.21.15.78:0/1186515069 >> 172.21.15.78:0/1186515069 conn(0x7f9a040742a0 msgr2=0x7f9a040667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:48.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.454+0000 7f9a0a7ee700 1 -- 172.21.15.78:0/1186515069 shutdown_connections 2023-10-14T11:58:48.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.454+0000 7f9a0a7ee700 1 -- 172.21.15.78:0/1186515069 wait complete. 2023-10-14T11:58:48.468 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T11:58:48.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.589+0000 7f51cf5f4700 1 Processor -- start 2023-10-14T11:58:48.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.589+0000 7f51cf5f4700 1 -- start start 2023-10-14T11:58:48.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.590+0000 7f51cf5f4700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f51c813dbe0 0x7f51c806a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:48.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.590+0000 7f51cf5f4700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f51c80c4540 0x7f51c80c02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:48.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.590+0000 7f51cf5f4700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51c806ebe0 con 0x7f51c806ae10 2023-10-14T11:58:48.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.590+0000 7f51cf5f4700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51c806d330 con 0x7f51c813dbe0 2023-10-14T11:58:48.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.590+0000 7f51cf5f4700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51c81558c0 con 0x7f51c80c4540 2023-10-14T11:58:48.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.590+0000 7f51cd390700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f51c813dbe0 0x7f51c806a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:48.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.590+0000 7f51cdb91700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f51c806ae10 0x7f51c80c0820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50688/0 (socket says 172.21.15.78:50688) 2023-10-14T11:58:48.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.590+0000 7f51cd390700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f51c813dbe0 0x7f51c806a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45810/0 (socket says 172.21.15.78:45810) 2023-10-14T11:58:48.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.590+0000 7f51cdb91700 1 -- 172.21.15.78:0/1951550185 learned_addr learned my addr 172.21.15.78:0/1951550185 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:48.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.590+0000 7f51ccb8f700 1 --2- 172.21.15.78:0/1951550185 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f51c80c4540 0x7f51c80c02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:48.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.591+0000 7f51cd390700 1 -- 172.21.15.78:0/1951550185 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f51c80c4540 msgr2=0x7f51c80c02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:48.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.591+0000 7f51cd390700 1 --2- 172.21.15.78:0/1951550185 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f51c80c4540 0x7f51c80c02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:48.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.591+0000 7f51cd390700 1 -- 172.21.15.78:0/1951550185 >> v1:172.21.15.78:6789/0 conn(0x7f51c806ae10 legacy=0x7f51c80c0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:48.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.591+0000 7f51cd390700 1 -- 172.21.15.78:0/1951550185 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c8154b00 con 0x7f51c813dbe0 2023-10-14T11:58:48.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.591+0000 7f51bffff700 1 -- 172.21.15.78:0/1951550185 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1622742472 0 0) 0x7f51c806ebe0 con 0x7f51c806ae10 2023-10-14T11:58:48.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.591+0000 7f51cd390700 1 --2- 172.21.15.78:0/1951550185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51c813dbe0 0x7f51c806a8d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f51b8009840 tx=0x7f51b8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37c533c98881d140 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:48.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.591+0000 7f51bffff700 1 -- 172.21.15.78:0/1951550185 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b8011830 con 0x7f51c813dbe0 2023-10-14T11:58:48.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.591+0000 7f51bffff700 1 -- 172.21.15.78:0/1951550185 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51b8011990 con 0x7f51c813dbe0 2023-10-14T11:58:48.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.591+0000 7f51bffff700 1 -- 172.21.15.78:0/1951550185 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b8011d20 con 0x7f51c813dbe0 2023-10-14T11:58:48.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.591+0000 7f51cf5f4700 1 -- 172.21.15.78:0/1951550185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51c813dbe0 msgr2=0x7f51c806a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:48.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.591+0000 7f51cf5f4700 1 --2- 172.21.15.78:0/1951550185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51c813dbe0 0x7f51c806a8d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f51b8009840 tx=0x7f51b8009b50 comp rx=0 tx=0).stop 2023-10-14T11:58:48.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.591+0000 7f51cf5f4700 1 -- 172.21.15.78:0/1951550185 shutdown_connections 2023-10-14T11:58:48.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.591+0000 7f51cf5f4700 1 --2- 172.21.15.78:0/1951550185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51c813dbe0 0x7f51c806a8d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:48.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.591+0000 7f51cf5f4700 1 --2- 172.21.15.78:0/1951550185 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f51c80c4540 0x7f51c80c02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:48.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.591+0000 7f51cf5f4700 1 -- 172.21.15.78:0/1951550185 >> 172.21.15.78:0/1951550185 conn(0x7f51c80c3a50 msgr2=0x7f51c80c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:48.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.592+0000 7f51cf5f4700 1 -- 172.21.15.78:0/1951550185 shutdown_connections 2023-10-14T11:58:48.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.592+0000 7f51cf5f4700 1 -- 172.21.15.78:0/1951550185 wait complete. 2023-10-14T11:58:48.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.593+0000 7f51cf5f4700 1 Processor -- start 2023-10-14T11:58:48.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.593+0000 7f51cf5f4700 1 -- start start 2023-10-14T11:58:48.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.593+0000 7f51cf5f4700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f51c806ae10 0x7f51c8158970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:48.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.593+0000 7f51cf5f4700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f51c80c4540 0x7f51c80c6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:48.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.593+0000 7f51cf5f4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51c813dbe0 0x7f51c80c0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:48.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.593+0000 7f51cf5f4700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c8156850 con 0x7f51c813dbe0 2023-10-14T11:58:48.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.593+0000 7f51cf5f4700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c80c2d10 con 0x7f51c80c4540 2023-10-14T11:58:48.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.593+0000 7f51cf5f4700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51c80c6d00 con 0x7f51c806ae10 2023-10-14T11:58:48.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.593+0000 7f51cdb91700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51c813dbe0 0x7f51c80c0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:48.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.593+0000 7f51cdb91700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51c813dbe0 0x7f51c80c0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45818/0 (socket says 172.21.15.78:45818) 2023-10-14T11:58:48.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.593+0000 7f51cdb91700 1 -- 172.21.15.78:0/1729778881 learned_addr learned my addr 172.21.15.78:0/1729778881 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:48.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.594+0000 7f51ccb8f700 1 --2- 172.21.15.78:0/1729778881 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f51c80c4540 0x7f51c80c6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:48.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.594+0000 7f51cdb91700 1 -- 172.21.15.78:0/1729778881 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f51c806ae10 msgr2=0x7f51c8158970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:48.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.594+0000 7f51cd390700 1 --2- 172.21.15.78:0/1729778881 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f51c806ae10 0x7f51c8158970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:48.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.594+0000 7f51cdb91700 1 --2- 172.21.15.78:0/1729778881 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f51c806ae10 0x7f51c8158970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:48.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.594+0000 7f51cdb91700 1 -- 172.21.15.78:0/1729778881 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f51c80c4540 msgr2=0x7f51c80c6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:48.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.594+0000 7f51cdb91700 1 --2- 172.21.15.78:0/1729778881 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f51c80c4540 0x7f51c80c6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:48.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.594+0000 7f51cdb91700 1 -- 172.21.15.78:0/1729778881 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51b80094a0 con 0x7f51c813dbe0 2023-10-14T11:58:48.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.594+0000 7f51ccb8f700 1 --2- 172.21.15.78:0/1729778881 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f51c80c4540 0x7f51c80c6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:48.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.594+0000 7f51cd390700 1 --2- 172.21.15.78:0/1729778881 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f51c806ae10 0x7f51c8158970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:48.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.594+0000 7f51cdb91700 1 --2- 172.21.15.78:0/1729778881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51c813dbe0 0x7f51c80c0d20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f51c000cb90 tx=0x7f51c000cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:48.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.594+0000 7f51bdffb700 1 -- 172.21.15.78:0/1729778881 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c000ab70 con 0x7f51c813dbe0 2023-10-14T11:58:48.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.594+0000 7f51bdffb700 1 -- 172.21.15.78:0/1729778881 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c000acf0 con 0x7f51c813dbe0 2023-10-14T11:58:48.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.594+0000 7f51cf5f4700 1 -- 172.21.15.78:0/1729778881 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51c8155f20 con 0x7f51c813dbe0 2023-10-14T11:58:48.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.594+0000 7f51bdffb700 1 -- 172.21.15.78:0/1729778881 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c0007310 con 0x7f51c813dbe0 2023-10-14T11:58:48.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.594+0000 7f51cf5f4700 1 -- 172.21.15.78:0/1729778881 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51c81564e0 con 0x7f51c813dbe0 2023-10-14T11:58:48.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.595+0000 7f51cf5f4700 1 -- 172.21.15.78:0/1729778881 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51c8074b80 con 0x7f51c813dbe0 2023-10-14T11:58:48.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.596+0000 7f51bdffb700 1 -- 172.21.15.78:0/1729778881 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f51c0007470 con 0x7f51c813dbe0 2023-10-14T11:58:48.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.596+0000 7f51bdffb700 1 --2- 172.21.15.78:0/1729778881 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f51ac06af00 0x7f51ac06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:48.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.596+0000 7f51bdffb700 1 -- 172.21.15.78:0/1729778881 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f51c0089060 con 0x7f51c813dbe0 2023-10-14T11:58:48.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.596+0000 7f51cd390700 1 --2- 172.21.15.78:0/1729778881 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f51ac06af00 0x7f51ac06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:48.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.597+0000 7f51cd390700 1 --2- 172.21.15.78:0/1729778881 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f51ac06af00 0x7f51ac06d3c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f51b80090b0 tx=0x7f51b8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:48.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.600+0000 7f51bdffb700 1 -- 172.21.15.78:0/1729778881 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f51c0054110 con 0x7f51c813dbe0 2023-10-14T11:58:48.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.790+0000 7f51cf5f4700 1 -- 172.21.15.78:0/1729778881 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f51c806b2e0 con 0x7f51c813dbe0 2023-10-14T11:58:48.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.790+0000 7f51bdffb700 1 -- 172.21.15.78:0/1729778881 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f51c0018020 con 0x7f51c813dbe0 2023-10-14T11:58:48.792 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2023-10-14T11:58:48.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.792+0000 7f51cf5f4700 1 -- 172.21.15.78:0/1729778881 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f51ac06af00 msgr2=0x7f51ac06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:48.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.792+0000 7f51cf5f4700 1 --2- 172.21.15.78:0/1729778881 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f51ac06af00 0x7f51ac06d3c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f51b80090b0 tx=0x7f51b8008030 comp rx=0 tx=0).stop 2023-10-14T11:58:48.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.792+0000 7f51cf5f4700 1 -- 172.21.15.78:0/1729778881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51c813dbe0 msgr2=0x7f51c80c0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:48.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.792+0000 7f51cf5f4700 1 --2- 172.21.15.78:0/1729778881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51c813dbe0 0x7f51c80c0d20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f51c000cb90 tx=0x7f51c000cf50 comp rx=0 tx=0).stop 2023-10-14T11:58:48.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.792+0000 7f51cf5f4700 1 -- 172.21.15.78:0/1729778881 shutdown_connections 2023-10-14T11:58:48.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.792+0000 7f51cf5f4700 1 --2- 172.21.15.78:0/1729778881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51c813dbe0 0x7f51c80c0d20 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:48.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.792+0000 7f51cf5f4700 1 --2- 172.21.15.78:0/1729778881 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f51c806ae10 0x7f51c8158970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:48.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.792+0000 7f51cf5f4700 1 --2- 172.21.15.78:0/1729778881 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f51ac06af00 0x7f51ac06d3c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:48.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.792+0000 7f51cf5f4700 1 --2- 172.21.15.78:0/1729778881 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f51c80c4540 0x7f51c80c6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:48.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.792+0000 7f51cf5f4700 1 -- 172.21.15.78:0/1729778881 >> 172.21.15.78:0/1729778881 conn(0x7f51c80c3a50 msgr2=0x7f51c80c2260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:48.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.792+0000 7f51cf5f4700 1 -- 172.21.15.78:0/1729778881 shutdown_connections 2023-10-14T11:58:48.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.792+0000 7f51cf5f4700 1 -- 172.21.15.78:0/1729778881 wait complete. 2023-10-14T11:58:48.806 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T11:58:48.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.924+0000 7f1e72265700 1 Processor -- start 2023-10-14T11:58:48.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.924+0000 7f1e72265700 1 -- start start 2023-10-14T11:58:48.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.925+0000 7f1e72265700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1e6c0c3280 0x7f1e6c151c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:48.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.925+0000 7f1e72265700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1e6c1521a0 0x7f1e6c158630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:48.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.925+0000 7f1e72265700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e6c158b70 con 0x7f1e6c0c45b0 2023-10-14T11:58:48.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.925+0000 7f1e72265700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e6c06d960 con 0x7f1e6c0c3280 2023-10-14T11:58:48.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.925+0000 7f1e72265700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6c06d600 con 0x7f1e6c1521a0 2023-10-14T11:58:48.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.925+0000 7f1e6bfff700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1e6c0c3280 0x7f1e6c151c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:48.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.925+0000 7f1e6bfff700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1e6c0c3280 0x7f1e6c151c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45822/0 (socket says 172.21.15.78:45822) 2023-10-14T11:58:48.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.925+0000 7f1e6bfff700 1 -- 172.21.15.78:0/664574241 learned_addr learned my addr 172.21.15.78:0/664574241 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:48.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.926+0000 7f1e6bfff700 1 -- 172.21.15.78:0/664574241 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1e6c1521a0 msgr2=0x7f1e6c158630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:48.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.926+0000 7f1e6bfff700 1 --2- 172.21.15.78:0/664574241 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1e6c1521a0 0x7f1e6c158630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:48.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.926+0000 7f1e6bfff700 1 -- 172.21.15.78:0/664574241 >> v1:172.21.15.195:6789/0 conn(0x7f1e6c0c45b0 legacy=0x7f1e6c157f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:48.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.926+0000 7f1e6bfff700 1 -- 172.21.15.78:0/664574241 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e6c072560 con 0x7f1e6c0c3280 2023-10-14T11:58:48.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.926+0000 7f1e6bfff700 1 --2- 172.21.15.78:0/664574241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e6c0c3280 0x7f1e6c151c60 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1e600098f0 tx=0x7f1e60009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2c953eb34264782 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:48.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.926+0000 7f1e6affd700 1 -- 172.21.15.78:0/664574241 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e60004230 con 0x7f1e6c0c3280 2023-10-14T11:58:48.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.926+0000 7f1e6affd700 1 -- 172.21.15.78:0/664574241 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e60004390 con 0x7f1e6c0c3280 2023-10-14T11:58:48.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.926+0000 7f1e6affd700 1 -- 172.21.15.78:0/664574241 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e600036a0 con 0x7f1e6c0c3280 2023-10-14T11:58:48.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.926+0000 7f1e72265700 1 -- 172.21.15.78:0/664574241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e6c0c3280 msgr2=0x7f1e6c151c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:48.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.926+0000 7f1e72265700 1 --2- 172.21.15.78:0/664574241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e6c0c3280 0x7f1e6c151c60 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1e600098f0 tx=0x7f1e60009c00 comp rx=0 tx=0).stop 2023-10-14T11:58:48.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.926+0000 7f1e72265700 1 -- 172.21.15.78:0/664574241 shutdown_connections 2023-10-14T11:58:48.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.926+0000 7f1e72265700 1 --2- 172.21.15.78:0/664574241 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1e6c1521a0 0x7f1e6c158630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:48.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.926+0000 7f1e72265700 1 --2- 172.21.15.78:0/664574241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e6c0c3280 0x7f1e6c151c60 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:48.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.926+0000 7f1e72265700 1 -- 172.21.15.78:0/664574241 >> 172.21.15.78:0/664574241 conn(0x7f1e6c0c3a90 msgr2=0x7f1e6c0c3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:48.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.927+0000 7f1e72265700 1 -- 172.21.15.78:0/664574241 shutdown_connections 2023-10-14T11:58:48.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.927+0000 7f1e72265700 1 -- 172.21.15.78:0/664574241 wait complete. 2023-10-14T11:58:48.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.928+0000 7f1e72265700 1 Processor -- start 2023-10-14T11:58:48.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.928+0000 7f1e72265700 1 -- start start 2023-10-14T11:58:48.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.928+0000 7f1e72265700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e6c0c3280 0x7f1e6c14bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:48.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.928+0000 7f1e72265700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1e6c0c45b0 0x7f1e6c154030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:48.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.928+0000 7f1e72265700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1e6c1521a0 0x7f1e6c1e6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:48.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.929+0000 7f1e72265700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6c0c66f0 con 0x7f1e6c0c3280 2023-10-14T11:58:48.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.929+0000 7f1e72265700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6c0c6900 con 0x7f1e6c1521a0 2023-10-14T11:58:48.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.929+0000 7f1e72265700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6c1e9e60 con 0x7f1e6c0c45b0 2023-10-14T11:58:48.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.929+0000 7f1e6bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e6c0c3280 0x7f1e6c14bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:48.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.929+0000 7f1e6bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e6c0c3280 0x7f1e6c14bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45828/0 (socket says 172.21.15.78:45828) 2023-10-14T11:58:48.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.929+0000 7f1e6bfff700 1 -- 172.21.15.78:0/3731723875 learned_addr learned my addr 172.21.15.78:0/3731723875 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:48.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.929+0000 7f1e6b7fe700 1 --2- 172.21.15.78:0/3731723875 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1e6c0c45b0 0x7f1e6c154030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:48.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.929+0000 7f1e70802700 1 --2- 172.21.15.78:0/3731723875 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1e6c1521a0 0x7f1e6c1e6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:48.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.929+0000 7f1e6bfff700 1 -- 172.21.15.78:0/3731723875 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1e6c0c45b0 msgr2=0x7f1e6c154030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:48.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.929+0000 7f1e6bfff700 1 --2- 172.21.15.78:0/3731723875 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1e6c0c45b0 0x7f1e6c154030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:48.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.929+0000 7f1e6bfff700 1 -- 172.21.15.78:0/3731723875 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1e6c1521a0 msgr2=0x7f1e6c1e6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:48.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.929+0000 7f1e6bfff700 1 --2- 172.21.15.78:0/3731723875 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1e6c1521a0 0x7f1e6c1e6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:48.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.929+0000 7f1e6bfff700 1 -- 172.21.15.78:0/3731723875 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e60009550 con 0x7f1e6c0c3280 2023-10-14T11:58:48.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.929+0000 7f1e6bfff700 1 --2- 172.21.15.78:0/3731723875 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e6c0c3280 0x7f1e6c14bea0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c073030 tx=0x7f1e60004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:48.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.929+0000 7f1e68ff9700 1 -- 172.21.15.78:0/3731723875 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e6000b860 con 0x7f1e6c0c3280 2023-10-14T11:58:48.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.929+0000 7f1e68ff9700 1 -- 172.21.15.78:0/3731723875 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e6000ba00 con 0x7f1e6c0c3280 2023-10-14T11:58:48.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.929+0000 7f1e72265700 1 -- 172.21.15.78:0/3731723875 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e6c1ea140 con 0x7f1e6c0c3280 2023-10-14T11:58:48.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.929+0000 7f1e68ff9700 1 -- 172.21.15.78:0/3731723875 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e6000bdb0 con 0x7f1e6c0c3280 2023-10-14T11:58:48.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.929+0000 7f1e72265700 1 -- 172.21.15.78:0/3731723875 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e6c154570 con 0x7f1e6c0c3280 2023-10-14T11:58:48.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.930+0000 7f1e72265700 1 -- 172.21.15.78:0/3731723875 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e6c074b80 con 0x7f1e6c0c3280 2023-10-14T11:58:48.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.931+0000 7f1e68ff9700 1 -- 172.21.15.78:0/3731723875 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1e6001b440 con 0x7f1e6c0c3280 2023-10-14T11:58:48.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.931+0000 7f1e68ff9700 1 --2- 172.21.15.78:0/3731723875 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f1e5806af00 0x7f1e5806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:48.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.931+0000 7f1e68ff9700 1 -- 172.21.15.78:0/3731723875 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1e6008ae80 con 0x7f1e6c0c3280 2023-10-14T11:58:48.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.931+0000 7f1e6b7fe700 1 --2- 172.21.15.78:0/3731723875 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f1e5806af00 0x7f1e5806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:48.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.931+0000 7f1e6b7fe700 1 --2- 172.21.15.78:0/3731723875 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f1e5806af00 0x7f1e5806d3c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c14c3e0 tx=0x7f1e5c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:48.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:48.935+0000 7f1e68ff9700 1 -- 172.21.15.78:0/3731723875 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1e600560e0 con 0x7f1e6c0c3280 2023-10-14T11:58:49.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.166+0000 7f1e72265700 1 -- 172.21.15.78:0/3731723875 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e6c146d70 con 0x7f1e6c0c3280 2023-10-14T11:58:49.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.167+0000 7f1e68ff9700 1 -- 172.21.15.78:0/3731723875 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4904 (secure 0 0 0) 0x7f1e60059700 con 0x7f1e6c0c3280 2023-10-14T11:58:49.169 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:49.169 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/353043244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":353043244},{"type":"v1","addr":"172.21.15.78:6836","nonce":353043244}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6834/3389817343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":3389817343},{"type":"v1","addr":"172.21.15.195:6834","nonce":3389817343}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4267,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/460176701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6833","nonce":460176701},{"type":"v1","addr":"172.21.15.195:6835","nonce":460176701}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T11:58:14.535101+0000","modified":"2023-10-14T11:58:17.559202+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/920300787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":920300787},{"type":"v1","addr":"172.21.15.78:6837","nonce":920300787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-14T11:58:49.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.169+0000 7f1e72265700 1 -- 172.21.15.78:0/3731723875 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f1e5806af00 msgr2=0x7f1e5806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:49.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.169+0000 7f1e72265700 1 --2- 172.21.15.78:0/3731723875 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f1e5806af00 0x7f1e5806d3c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c14c3e0 tx=0x7f1e5c009040 comp rx=0 tx=0).stop 2023-10-14T11:58:49.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.169+0000 7f1e72265700 1 -- 172.21.15.78:0/3731723875 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e6c0c3280 msgr2=0x7f1e6c14bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:49.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.169+0000 7f1e72265700 1 --2- 172.21.15.78:0/3731723875 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e6c0c3280 0x7f1e6c14bea0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c073030 tx=0x7f1e60004750 comp rx=0 tx=0).stop 2023-10-14T11:58:49.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.169+0000 7f1e72265700 1 -- 172.21.15.78:0/3731723875 shutdown_connections 2023-10-14T11:58:49.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.169+0000 7f1e72265700 1 --2- 172.21.15.78:0/3731723875 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f1e5806af00 0x7f1e5806d3c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:49.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.169+0000 7f1e72265700 1 --2- 172.21.15.78:0/3731723875 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1e6c1521a0 0x7f1e6c1e6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:49.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.169+0000 7f1e72265700 1 --2- 172.21.15.78:0/3731723875 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e6c0c3280 0x7f1e6c14bea0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:49.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.169+0000 7f1e72265700 1 --2- 172.21.15.78:0/3731723875 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1e6c0c45b0 0x7f1e6c154030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:49.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.169+0000 7f1e72265700 1 -- 172.21.15.78:0/3731723875 >> 172.21.15.78:0/3731723875 conn(0x7f1e6c0c3a90 msgr2=0x7f1e6c147290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:49.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.169+0000 7f1e72265700 1 -- 172.21.15.78:0/3731723875 shutdown_connections 2023-10-14T11:58:49.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.169+0000 7f1e72265700 1 -- 172.21.15.78:0/3731723875 wait complete. 2023-10-14T11:58:49.174 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 8 2023-10-14T11:58:49.183 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T11:58:49.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.300+0000 7fc604e37700 1 Processor -- start 2023-10-14T11:58:49.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.300+0000 7fc604e37700 1 -- start start 2023-10-14T11:58:49.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.301+0000 7fc604e37700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc60006ab40 0x7fc60006f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:49.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.301+0000 7fc604e37700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fc6001554e0 0x7fc60006af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:49.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.301+0000 7fc604e37700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6000c49f0 con 0x7fc6000c0350 2023-10-14T11:58:49.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.301+0000 7fc604e37700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6000726a0 con 0x7fc60006ab40 2023-10-14T11:58:49.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.301+0000 7fc604e37700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc600072980 con 0x7fc6001554e0 2023-10-14T11:58:49.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.301+0000 7fc5fe59c700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc60006ab40 0x7fc60006f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:49.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.301+0000 7fc5fed9d700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc6000c0350 0x7fc60013d7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50716/0 (socket says 172.21.15.78:50716) 2023-10-14T11:58:49.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.301+0000 7fc5fed9d700 1 -- 172.21.15.78:0/692951755 learned_addr learned my addr 172.21.15.78:0/692951755 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:49.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.301+0000 7fc5fdd9b700 1 --2- 172.21.15.78:0/692951755 >> v2:172.21.15.195:3300/0 conn(0x7fc6001554e0 0x7fc60006af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:49.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.301+0000 7fc5fd59a700 1 -- 172.21.15.78:0/692951755 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2237084237 0 0) 0x7fc6000c49f0 con 0x7fc6000c0350 2023-10-14T11:58:49.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.302+0000 7fc5fd59a700 1 -- 172.21.15.78:0/692951755 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5e80035b0 con 0x7fc6000c0350 2023-10-14T11:58:49.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.302+0000 7fc5fe59c700 1 -- 172.21.15.78:0/692951755 >> v2:172.21.15.195:3300/0 conn(0x7fc6001554e0 msgr2=0x7fc60006af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:49.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.302+0000 7fc5fe59c700 1 --2- 172.21.15.78:0/692951755 >> v2:172.21.15.195:3300/0 conn(0x7fc6001554e0 0x7fc60006af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:49.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.302+0000 7fc5fe59c700 1 -- 172.21.15.78:0/692951755 >> v1:172.21.15.78:6789/0 conn(0x7fc6000c0350 legacy=0x7fc60013d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:49.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.302+0000 7fc5fe59c700 1 -- 172.21.15.78:0/692951755 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc600154a00 con 0x7fc60006ab40 2023-10-14T11:58:49.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.302+0000 7fc5fe59c700 1 --2- 172.21.15.78:0/692951755 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc60006ab40 0x7fc60006f1e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc60006ff00 tx=0x7fc5f4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cc2ef0990ada33c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:49.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.302+0000 7fc5fd59a700 1 -- 172.21.15.78:0/692951755 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f4011830 con 0x7fc60006ab40 2023-10-14T11:58:49.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.302+0000 7fc5fd59a700 1 -- 172.21.15.78:0/692951755 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5f4011990 con 0x7fc60006ab40 2023-10-14T11:58:49.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.302+0000 7fc5fd59a700 1 -- 172.21.15.78:0/692951755 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f4011d20 con 0x7fc60006ab40 2023-10-14T11:58:49.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.302+0000 7fc604e37700 1 -- 172.21.15.78:0/692951755 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc60006ab40 msgr2=0x7fc60006f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:49.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.302+0000 7fc604e37700 1 --2- 172.21.15.78:0/692951755 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc60006ab40 0x7fc60006f1e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc60006ff00 tx=0x7fc5f4009b20 comp rx=0 tx=0).stop 2023-10-14T11:58:49.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.302+0000 7fc604e37700 1 -- 172.21.15.78:0/692951755 shutdown_connections 2023-10-14T11:58:49.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.302+0000 7fc604e37700 1 --2- 172.21.15.78:0/692951755 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc60006ab40 0x7fc60006f1e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:49.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.302+0000 7fc604e37700 1 --2- 172.21.15.78:0/692951755 >> v2:172.21.15.195:3300/0 conn(0x7fc6001554e0 0x7fc60006af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:49.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.302+0000 7fc604e37700 1 -- 172.21.15.78:0/692951755 >> 172.21.15.78:0/692951755 conn(0x7fc6000c3ac0 msgr2=0x7fc6000c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:49.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.303+0000 7fc604e37700 1 -- 172.21.15.78:0/692951755 shutdown_connections 2023-10-14T11:58:49.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.303+0000 7fc604e37700 1 -- 172.21.15.78:0/692951755 wait complete. 2023-10-14T11:58:49.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.304+0000 7fc604e37700 1 Processor -- start 2023-10-14T11:58:49.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.304+0000 7fc604e37700 1 -- start start 2023-10-14T11:58:49.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.304+0000 7fc604e37700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc60006ab40 0x7fc600153370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:49.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.304+0000 7fc604e37700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc6000c0350 0x7fc6001369e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:49.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.304+0000 7fc604e37700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6001554e0 0x7fc600136f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:49.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.304+0000 7fc604e37700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6001ecd20 con 0x7fc6001554e0 2023-10-14T11:58:49.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.304+0000 7fc604e37700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6001edb80 con 0x7fc6000c0350 2023-10-14T11:58:49.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.304+0000 7fc604e37700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6001ecf70 con 0x7fc60006ab40 2023-10-14T11:58:49.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.305+0000 7fc5fed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6001554e0 0x7fc600136f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:49.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.305+0000 7fc5fed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6001554e0 0x7fc600136f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45846/0 (socket says 172.21.15.78:45846) 2023-10-14T11:58:49.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.305+0000 7fc5fed9d700 1 -- 172.21.15.78:0/1269545271 learned_addr learned my addr 172.21.15.78:0/1269545271 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:49.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.305+0000 7fc5fe59c700 1 --2- 172.21.15.78:0/1269545271 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc60006ab40 0x7fc600153370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:49.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.305+0000 7fc5fdd9b700 1 --2- 172.21.15.78:0/1269545271 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc6000c0350 0x7fc6001369e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:49.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.305+0000 7fc5fed9d700 1 -- 172.21.15.78:0/1269545271 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc60006ab40 msgr2=0x7fc600153370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:49.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.305+0000 7fc5fed9d700 1 --2- 172.21.15.78:0/1269545271 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc60006ab40 0x7fc600153370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:49.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.305+0000 7fc5fed9d700 1 -- 172.21.15.78:0/1269545271 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc6000c0350 msgr2=0x7fc6001369e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:49.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.305+0000 7fc5fed9d700 1 --2- 172.21.15.78:0/1269545271 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc6000c0350 0x7fc6001369e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:49.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.305+0000 7fc5fed9d700 1 -- 172.21.15.78:0/1269545271 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f40094a0 con 0x7fc6001554e0 2023-10-14T11:58:49.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.305+0000 7fc5fed9d700 1 --2- 172.21.15.78:0/1269545271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6001554e0 0x7fc600136f20 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc5f800cb70 tx=0x7fc5f800ce80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:49.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.305+0000 7fc5eeffd700 1 -- 172.21.15.78:0/1269545271 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f800aa60 con 0x7fc6001554e0 2023-10-14T11:58:49.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.305+0000 7fc5eeffd700 1 -- 172.21.15.78:0/1269545271 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5f800abc0 con 0x7fc6001554e0 2023-10-14T11:58:49.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.305+0000 7fc604e37700 1 -- 172.21.15.78:0/1269545271 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6001ed250 con 0x7fc6001554e0 2023-10-14T11:58:49.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.305+0000 7fc5eeffd700 1 -- 172.21.15.78:0/1269545271 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f8012400 con 0x7fc6001554e0 2023-10-14T11:58:49.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.305+0000 7fc604e37700 1 -- 172.21.15.78:0/1269545271 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6001ed810 con 0x7fc6001554e0 2023-10-14T11:58:49.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.306+0000 7fc604e37700 1 -- 172.21.15.78:0/1269545271 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc600074b80 con 0x7fc6001554e0 2023-10-14T11:58:49.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.307+0000 7fc5eeffd700 1 -- 172.21.15.78:0/1269545271 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc5f801b3b0 con 0x7fc6001554e0 2023-10-14T11:58:49.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.307+0000 7fc5eeffd700 1 --2- 172.21.15.78:0/1269545271 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc5e006af00 0x7fc5e006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:49.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.307+0000 7fc5eeffd700 1 -- 172.21.15.78:0/1269545271 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc5f80891c0 con 0x7fc6001554e0 2023-10-14T11:58:49.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.307+0000 7fc5fe59c700 1 --2- 172.21.15.78:0/1269545271 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc5e006af00 0x7fc5e006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:49.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.307+0000 7fc5fe59c700 1 --2- 172.21.15.78:0/1269545271 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc5e006af00 0x7fc5e006d3c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc5f40090b0 tx=0x7fc5f4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:49.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.311+0000 7fc5eeffd700 1 -- 172.21.15.78:0/1269545271 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc5f8054300 con 0x7fc6001554e0 2023-10-14T11:58:49.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.503+0000 7fc604e37700 1 -- 172.21.15.78:0/1269545271 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc6000ac4d0 con 0x7fc6001554e0 2023-10-14T11:58:49.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.504+0000 7fc5eeffd700 1 -- 172.21.15.78:0/1269545271 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16481 (secure 0 0 0) 0x7fc5f8057920 con 0x7fc6001554e0 2023-10-14T11:58:49.506 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:49.506 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":21,"fsid":"4c5dc134-3092-406a-a1e1-13e6d9e926e8","created":"2023-10-14T11:57:57.363750+0000","modified":"2023-10-14T11:58:16.139413+0000","last_up_change":"2023-10-14T11:58:06.982472+0000","last_in_change":"2023-10-14T11:57:59.821373+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T11:58:09.001200+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T11:58:12.778573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T11:58:13.490286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"625a5fc0-9474-4eae-b80e-14c27a74ad19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6801","nonce":3884969780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6803","nonce":3884969780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6807","nonce":3884969780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6805","nonce":3884969780}]},"public_addr":"172.21.15.78:6801/3884969780","cluster_addr":"172.21.15.78:6803/3884969780","heartbeat_back_addr":"172.21.15.78:6807/3884969780","heartbeat_front_addr":"172.21.15.78:6805/3884969780","state":["exists","up"]},{"osd":1,"uuid":"49d63db1-2981-49ab-9c40-c50fc744b982","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6809","nonce":2176928406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6811","nonce":2176928406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6815","nonce":2176928406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6813","nonce":2176928406}]},"public_addr":"172.21.15.78:6809/2176928406","cluster_addr":"172.21.15.78:6811/2176928406","heartbeat_back_addr":"172.21.15.78:6815/2176928406","heartbeat_front_addr":"172.21.15.78:6813/2176928406","state":["exists","up"]},{"osd":2,"uuid":"1d7d0b29-cbd1-4717-9fe6-765a8d4998b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6825","nonce":3647044577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6827","nonce":3647044577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6831","nonce":3647044577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6829","nonce":3647044577}]},"public_addr":"172.21.15.78:6825/3647044577","cluster_addr":"172.21.15.78:6827/3647044577","heartbeat_back_addr":"172.21.15.78:6831/3647044577","heartbeat_front_addr":"172.21.15.78:6829/3647044577","state":["exists","up"]},{"osd":3,"uuid":"2f75ded1-e9e2-40e7-99ae-45e1e425e1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6817","nonce":3584112288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6819","nonce":3584112288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6823","nonce":3584112288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6821","nonce":3584112288}]},"public_addr":"172.21.15.78:6817/3584112288","cluster_addr":"172.21.15.78:6819/3584112288","heartbeat_back_addr":"172.21.15.78:6823/3584112288","heartbeat_front_addr":"172.21.15.78:6821/3584112288","state":["exists","up"]},{"osd":4,"uuid":"5130b432-8358-46cb-8a31-3064c976f6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6801","nonce":3732756301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6803","nonce":3732756301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6807","nonce":3732756301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6805","nonce":3732756301}]},"public_addr":"172.21.15.195:6801/3732756301","cluster_addr":"172.21.15.195:6803/3732756301","heartbeat_back_addr":"172.21.15.195:6807/3732756301","heartbeat_front_addr":"172.21.15.195:6805/3732756301","state":["exists","up"]},{"osd":5,"uuid":"60e234a9-a506-4e40-9c19-49a58f3cf28c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6825","nonce":2857706411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6827","nonce":2857706411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6831","nonce":2857706411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6829","nonce":2857706411}]},"public_addr":"172.21.15.195:6825/2857706411","cluster_addr":"172.21.15.195:6827/2857706411","heartbeat_back_addr":"172.21.15.195:6831/2857706411","heartbeat_front_addr":"172.21.15.195:6829/2857706411","state":["exists","up"]},{"osd":6,"uuid":"df7d2dd6-26b6-4eb1-96e5-94d1f3955b33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6810","nonce":72554374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6813","nonce":72554374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6820","nonce":72554374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6817","nonce":72554374}]},"public_addr":"172.21.15.195:6810/72554374","cluster_addr":"172.21.15.195:6813/72554374","heartbeat_back_addr":"172.21.15.195:6820/72554374","heartbeat_front_addr":"172.21.15.195:6817/72554374","state":["exists","up"]},{"osd":7,"uuid":"47a9d4de-f56d-4aa4-9900-c4a9cf9b2f97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6809","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6811","nonce":482472979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6816","nonce":482472979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6823","nonce":482472979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6821","nonce":482472979}]},"public_addr":"172.21.15.195:6811/482472979","cluster_addr":"172.21.15.195:6816/482472979","heartbeat_back_addr":"172.21.15.195:6823/482472979","heartbeat_front_addr":"172.21.15.195:6821/482472979","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.754003+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.005994+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.043603+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.024506+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.062585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.033767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.994825+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.038300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T11:58:49.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.506+0000 7fc604e37700 1 -- 172.21.15.78:0/1269545271 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc5e006af00 msgr2=0x7fc5e006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:49.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.506+0000 7fc604e37700 1 --2- 172.21.15.78:0/1269545271 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc5e006af00 0x7fc5e006d3c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc5f40090b0 tx=0x7fc5f4008030 comp rx=0 tx=0).stop 2023-10-14T11:58:49.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.506+0000 7fc604e37700 1 -- 172.21.15.78:0/1269545271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6001554e0 msgr2=0x7fc600136f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:49.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.506+0000 7fc604e37700 1 --2- 172.21.15.78:0/1269545271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6001554e0 0x7fc600136f20 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc5f800cb70 tx=0x7fc5f800ce80 comp rx=0 tx=0).stop 2023-10-14T11:58:49.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.506+0000 7fc604e37700 1 -- 172.21.15.78:0/1269545271 shutdown_connections 2023-10-14T11:58:49.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.506+0000 7fc604e37700 1 --2- 172.21.15.78:0/1269545271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6001554e0 0x7fc600136f20 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:49.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.506+0000 7fc604e37700 1 --2- 172.21.15.78:0/1269545271 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc60006ab40 0x7fc600153370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:49.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.506+0000 7fc604e37700 1 --2- 172.21.15.78:0/1269545271 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc5e006af00 0x7fc5e006d3c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:49.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.506+0000 7fc604e37700 1 --2- 172.21.15.78:0/1269545271 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc6000c0350 0x7fc6001369e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:49.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.506+0000 7fc604e37700 1 -- 172.21.15.78:0/1269545271 >> 172.21.15.78:0/1269545271 conn(0x7fc6000c3ac0 msgr2=0x7fc6000c0750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:49.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.506+0000 7fc604e37700 1 -- 172.21.15.78:0/1269545271 shutdown_connections 2023-10-14T11:58:49.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.506+0000 7fc604e37700 1 -- 172.21.15.78:0/1269545271 wait complete. 2023-10-14T11:58:49.520 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-14T11:58:49.520 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T11:58:49.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.639+0000 7fa2e15ee700 1 Processor -- start 2023-10-14T11:58:49.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.639+0000 7fa2e15ee700 1 -- start start 2023-10-14T11:58:49.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.640+0000 7fa2e15ee700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa2dc154b90 0x7fa2dc0c5150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:49.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.640+0000 7fa2e15ee700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2dc0c9970 con 0x7fa2dc06cd00 2023-10-14T11:58:49.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.640+0000 7fa2e15ee700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2dc0c4e40 con 0x7fa2dc14b7e0 2023-10-14T11:58:49.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.640+0000 7fa2e15ee700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2dc0c81c0 con 0x7fa2dc154b90 2023-10-14T11:58:49.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.640+0000 7fa2da7fc700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa2dc154b90 0x7fa2dc0c5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:49.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.640+0000 7fa2db7fe700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fa2dc06cd00 0x7fa2dc06b440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50728/0 (socket says 172.21.15.78:50728) 2023-10-14T11:58:49.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.640+0000 7fa2db7fe700 1 -- 172.21.15.78:0/4258642431 learned_addr learned my addr 172.21.15.78:0/4258642431 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:49.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.640+0000 7fa2da7fc700 1 -- 172.21.15.78:0/4258642431 >> v1:172.21.15.195:6789/0 conn(0x7fa2dc14b7e0 legacy=0x7fa2dc0c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:49.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.640+0000 7fa2da7fc700 1 -- 172.21.15.78:0/4258642431 >> v1:172.21.15.78:6789/0 conn(0x7fa2dc06cd00 legacy=0x7fa2dc06b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:49.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.640+0000 7fa2da7fc700 1 -- 172.21.15.78:0/4258642431 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2dc0c84a0 con 0x7fa2dc154b90 2023-10-14T11:58:49.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.640+0000 7fa2da7fc700 1 --2- 172.21.15.78:0/4258642431 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2dc154b90 0x7fa2dc0c5150 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc009840 tx=0x7fa2cc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0179b6c5b5ee937 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:49.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.641+0000 7fa2d9ffb700 1 -- 172.21.15.78:0/4258642431 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2cc004230 con 0x7fa2dc154b90 2023-10-14T11:58:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.641+0000 7fa2d9ffb700 1 -- 172.21.15.78:0/4258642431 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2cc004390 con 0x7fa2dc154b90 2023-10-14T11:58:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.641+0000 7fa2d9ffb700 1 -- 172.21.15.78:0/4258642431 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2cc0036a0 con 0x7fa2dc154b90 2023-10-14T11:58:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.641+0000 7fa2e15ee700 1 -- 172.21.15.78:0/4258642431 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2dc154b90 msgr2=0x7fa2dc0c5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.641+0000 7fa2e15ee700 1 --2- 172.21.15.78:0/4258642431 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2dc154b90 0x7fa2dc0c5150 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc009840 tx=0x7fa2cc009b50 comp rx=0 tx=0).stop 2023-10-14T11:58:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.641+0000 7fa2e15ee700 1 -- 172.21.15.78:0/4258642431 shutdown_connections 2023-10-14T11:58:49.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.641+0000 7fa2e15ee700 1 --2- 172.21.15.78:0/4258642431 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2dc154b90 0x7fa2dc0c5150 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:49.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.641+0000 7fa2e15ee700 1 -- 172.21.15.78:0/4258642431 >> 172.21.15.78:0/4258642431 conn(0x7fa2dc0ca2c0 msgr2=0x7fa2dc0ca6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:49.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.641+0000 7fa2e15ee700 1 -- 172.21.15.78:0/4258642431 shutdown_connections 2023-10-14T11:58:49.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.641+0000 7fa2e15ee700 1 -- 172.21.15.78:0/4258642431 wait complete. 2023-10-14T11:58:49.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.642+0000 7fa2e15ee700 1 Processor -- start 2023-10-14T11:58:49.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.643+0000 7fa2e15ee700 1 -- start start 2023-10-14T11:58:49.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.643+0000 7fa2e15ee700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2dc06cd00 0x7fa2dc0aea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:49.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.643+0000 7fa2e15ee700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa2dc14b7e0 0x7fa2dc143d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:49.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.643+0000 7fa2e15ee700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa2dc154b90 0x7fa2dc0aacb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:49.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.643+0000 7fa2e15ee700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2dc0afd80 con 0x7fa2dc06cd00 2023-10-14T11:58:49.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.643+0000 7fa2e15ee700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2dc0aefa0 con 0x7fa2dc154b90 2023-10-14T11:58:49.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.643+0000 7fa2e15ee700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2dc144260 con 0x7fa2dc14b7e0 2023-10-14T11:58:49.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.643+0000 7fa2daffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2dc06cd00 0x7fa2dc0aea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:49.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.643+0000 7fa2daffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2dc06cd00 0x7fa2dc0aea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45856/0 (socket says 172.21.15.78:45856) 2023-10-14T11:58:49.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.643+0000 7fa2daffd700 1 -- 172.21.15.78:0/3983828545 learned_addr learned my addr 172.21.15.78:0/3983828545 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:49.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.643+0000 7fa2da7fc700 1 --2- 172.21.15.78:0/3983828545 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa2dc14b7e0 0x7fa2dc143d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:49.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.643+0000 7fa2db7fe700 1 --2- 172.21.15.78:0/3983828545 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa2dc154b90 0x7fa2dc0aacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:49.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.644+0000 7fa2daffd700 1 -- 172.21.15.78:0/3983828545 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa2dc14b7e0 msgr2=0x7fa2dc143d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:49.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.644+0000 7fa2daffd700 1 --2- 172.21.15.78:0/3983828545 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa2dc14b7e0 0x7fa2dc143d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:49.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.644+0000 7fa2daffd700 1 -- 172.21.15.78:0/3983828545 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa2dc154b90 msgr2=0x7fa2dc0aacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:49.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.644+0000 7fa2daffd700 1 --2- 172.21.15.78:0/3983828545 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa2dc154b90 0x7fa2dc0aacb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:49.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.644+0000 7fa2daffd700 1 -- 172.21.15.78:0/3983828545 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2cc0094a0 con 0x7fa2dc06cd00 2023-10-14T11:58:49.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.644+0000 7fa2daffd700 1 --2- 172.21.15.78:0/3983828545 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2dc06cd00 0x7fa2dc0aea60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa2d000c4d0 tx=0x7fa2d000c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:49.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.644+0000 7fa2c37fe700 1 -- 172.21.15.78:0/3983828545 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d0011790 con 0x7fa2dc06cd00 2023-10-14T11:58:49.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.644+0000 7fa2c37fe700 1 -- 172.21.15.78:0/3983828545 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2d00118f0 con 0x7fa2dc06cd00 2023-10-14T11:58:49.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.644+0000 7fa2e15ee700 1 -- 172.21.15.78:0/3983828545 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2dc156990 con 0x7fa2dc06cd00 2023-10-14T11:58:49.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.644+0000 7fa2c37fe700 1 -- 172.21.15.78:0/3983828545 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d0011ca0 con 0x7fa2dc06cd00 2023-10-14T11:58:49.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.644+0000 7fa2e15ee700 1 -- 172.21.15.78:0/3983828545 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2dc156f50 con 0x7fa2dc06cd00 2023-10-14T11:58:49.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.645+0000 7fa2e15ee700 1 -- 172.21.15.78:0/3983828545 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2dc0a9a80 con 0x7fa2dc06cd00 2023-10-14T11:58:49.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.645+0000 7fa2c37fe700 1 -- 172.21.15.78:0/3983828545 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa2d001c460 con 0x7fa2dc06cd00 2023-10-14T11:58:49.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.646+0000 7fa2c37fe700 1 --2- 172.21.15.78:0/3983828545 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa2c806af30 0x7fa2c806d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:49.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.646+0000 7fa2da7fc700 1 --2- 172.21.15.78:0/3983828545 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa2c806af30 0x7fa2c806d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:49.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.646+0000 7fa2c37fe700 1 -- 172.21.15.78:0/3983828545 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa2d0089240 con 0x7fa2dc06cd00 2023-10-14T11:58:49.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.646+0000 7fa2da7fc700 1 --2- 172.21.15.78:0/3983828545 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa2c806af30 0x7fa2c806d3f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc0090b0 tx=0x7fa2cc008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:49.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.650+0000 7fa2c37fe700 1 -- 172.21.15.78:0/3983828545 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa2d0054380 con 0x7fa2dc06cd00 2023-10-14T11:58:49.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.884+0000 7fa2e15ee700 1 -- 172.21.15.78:0/3983828545 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2dc074b80 con 0x7fa2dc06cd00 2023-10-14T11:58:49.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.884+0000 7fa2c37fe700 1 -- 172.21.15.78:0/3983828545 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4904 (secure 0 0 0) 0x7fa2d00579a0 con 0x7fa2dc06cd00 2023-10-14T11:58:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:49.887 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/353043244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":353043244},{"type":"v1","addr":"172.21.15.78:6836","nonce":353043244}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6834/3389817343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":3389817343},{"type":"v1","addr":"172.21.15.195:6834","nonce":3389817343}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4267,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/460176701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6833","nonce":460176701},{"type":"v1","addr":"172.21.15.195:6835","nonce":460176701}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T11:58:14.535101+0000","modified":"2023-10-14T11:58:17.559202+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/920300787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":920300787},{"type":"v1","addr":"172.21.15.78:6837","nonce":920300787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-14T11:58:49.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.886+0000 7fa2e15ee700 1 -- 172.21.15.78:0/3983828545 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa2c806af30 msgr2=0x7fa2c806d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:49.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.886+0000 7fa2e15ee700 1 --2- 172.21.15.78:0/3983828545 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa2c806af30 0x7fa2c806d3f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc0090b0 tx=0x7fa2cc008030 comp rx=0 tx=0).stop 2023-10-14T11:58:49.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.886+0000 7fa2e15ee700 1 -- 172.21.15.78:0/3983828545 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2dc06cd00 msgr2=0x7fa2dc0aea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:49.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.886+0000 7fa2e15ee700 1 --2- 172.21.15.78:0/3983828545 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2dc06cd00 0x7fa2dc0aea60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa2d000c4d0 tx=0x7fa2d000c7e0 comp rx=0 tx=0).stop 2023-10-14T11:58:49.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.886+0000 7fa2e15ee700 1 -- 172.21.15.78:0/3983828545 shutdown_connections 2023-10-14T11:58:49.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.886+0000 7fa2e15ee700 1 --2- 172.21.15.78:0/3983828545 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa2c806af30 0x7fa2c806d3f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:49.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.886+0000 7fa2e15ee700 1 --2- 172.21.15.78:0/3983828545 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa2dc154b90 0x7fa2dc0aacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:49.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.887+0000 7fa2e15ee700 1 --2- 172.21.15.78:0/3983828545 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2dc06cd00 0x7fa2dc0aea60 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:49.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.887+0000 7fa2e15ee700 1 --2- 172.21.15.78:0/3983828545 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa2dc14b7e0 0x7fa2dc143d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:49.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.887+0000 7fa2e15ee700 1 -- 172.21.15.78:0/3983828545 >> 172.21.15.78:0/3983828545 conn(0x7fa2dc0ca2c0 msgr2=0x7fa2dc0c5930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:49.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.887+0000 7fa2e15ee700 1 -- 172.21.15.78:0/3983828545 shutdown_connections 2023-10-14T11:58:49.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:49.887+0000 7fa2e15ee700 1 -- 172.21.15.78:0/3983828545 wait complete. 2023-10-14T11:58:49.892 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 8 2023-10-14T11:58:49.900 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T11:58:50.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.018+0000 7f6a0c334700 1 Processor -- start 2023-10-14T11:58:50.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.018+0000 7f6a0c334700 1 -- start start 2023-10-14T11:58:50.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.019+0000 7f6a0c334700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6a0406ab40 0x7f6a0406f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:50.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.019+0000 7f6a0c334700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6a040c0350 0x7f6a0413d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:50.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.019+0000 7f6a0c334700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a0406b630 con 0x7f6a041554e0 2023-10-14T11:58:50.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.019+0000 7f6a0c334700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a040c49f0 con 0x7f6a040c0350 2023-10-14T11:58:50.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.019+0000 7f6a0c334700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a040726a0 con 0x7f6a0406ab40 2023-10-14T11:58:50.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.019+0000 7f6a0a8d1700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6a040c0350 0x7f6a0413d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:50.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.019+0000 7f6a0a8d1700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6a040c0350 0x7f6a0413d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45868/0 (socket says 172.21.15.78:45868) 2023-10-14T11:58:50.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.019+0000 7f6a0a8d1700 1 -- 172.21.15.78:0/2046766025 learned_addr learned my addr 172.21.15.78:0/2046766025 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:50.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.019+0000 7f6a0a0d0700 1 --2- 172.21.15.78:0/2046766025 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6a0406ab40 0x7f6a0406f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:50.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.019+0000 7f6a0a8d1700 1 -- 172.21.15.78:0/2046766025 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6a0406ab40 msgr2=0x7f6a0406f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:50.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.019+0000 7f6a0a8d1700 1 --2- 172.21.15.78:0/2046766025 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6a0406ab40 0x7f6a0406f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:50.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.019+0000 7f6a0a8d1700 1 -- 172.21.15.78:0/2046766025 >> v1:172.21.15.195:6789/0 conn(0x7f6a041554e0 legacy=0x7f6a0406af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:50.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.019+0000 7f6a0a8d1700 1 -- 172.21.15.78:0/2046766025 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a04072980 con 0x7f6a040c0350 2023-10-14T11:58:50.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.020+0000 7f6a0a8d1700 1 --2- 172.21.15.78:0/2046766025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a040c0350 0x7f6a0413d7b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f69fc009e50 tx=0x7f69fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3938f62cb0a958b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:50.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.020+0000 7f6a090ce700 1 -- 172.21.15.78:0/2046766025 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69fc00e070 con 0x7f6a040c0350 2023-10-14T11:58:50.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.020+0000 7f6a090ce700 1 -- 172.21.15.78:0/2046766025 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69fc003a20 con 0x7f6a040c0350 2023-10-14T11:58:50.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.020+0000 7f6a090ce700 1 -- 172.21.15.78:0/2046766025 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69fc003dd0 con 0x7f6a040c0350 2023-10-14T11:58:50.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.020+0000 7f6a0c334700 1 -- 172.21.15.78:0/2046766025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a040c0350 msgr2=0x7f6a0413d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:50.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.020+0000 7f6a0c334700 1 --2- 172.21.15.78:0/2046766025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a040c0350 0x7f6a0413d7b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f69fc009e50 tx=0x7f69fc00b330 comp rx=0 tx=0).stop 2023-10-14T11:58:50.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.020+0000 7f6a0c334700 1 -- 172.21.15.78:0/2046766025 shutdown_connections 2023-10-14T11:58:50.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.020+0000 7f6a0c334700 1 --2- 172.21.15.78:0/2046766025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a040c0350 0x7f6a0413d7b0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:50.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.020+0000 7f6a0c334700 1 --2- 172.21.15.78:0/2046766025 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6a0406ab40 0x7f6a0406f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:50.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.020+0000 7f6a0c334700 1 -- 172.21.15.78:0/2046766025 >> 172.21.15.78:0/2046766025 conn(0x7f6a040c3ac0 msgr2=0x7f6a040c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:50.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.020+0000 7f6a0c334700 1 -- 172.21.15.78:0/2046766025 shutdown_connections 2023-10-14T11:58:50.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.020+0000 7f6a0c334700 1 -- 172.21.15.78:0/2046766025 wait complete. 2023-10-14T11:58:50.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.021+0000 7f6a0c334700 1 Processor -- start 2023-10-14T11:58:50.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.022+0000 7f6a0c334700 1 -- start start 2023-10-14T11:58:50.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.022+0000 7f6a0c334700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6a0406ab40 0x7f6a041ecfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:50.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.022+0000 7f6a0c334700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a040c0350 0x7f6a0414f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:50.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.022+0000 7f6a0c334700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6a041554e0 0x7f6a041577a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:50.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.022+0000 7f6a0c334700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a04151970 con 0x7f6a040c0350 2023-10-14T11:58:50.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.022+0000 7f6a0c334700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a04151c50 con 0x7f6a041554e0 2023-10-14T11:58:50.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.022+0000 7f6a0c334700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a041ed980 con 0x7f6a0406ab40 2023-10-14T11:58:50.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.022+0000 7f6a098cf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a040c0350 0x7f6a0414f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:50.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.022+0000 7f6a098cf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a040c0350 0x7f6a0414f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45872/0 (socket says 172.21.15.78:45872) 2023-10-14T11:58:50.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.022+0000 7f6a098cf700 1 -- 172.21.15.78:0/476567884 learned_addr learned my addr 172.21.15.78:0/476567884 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:50.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.022+0000 7f6a0a8d1700 1 --2- 172.21.15.78:0/476567884 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6a041554e0 0x7f6a041577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:50.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.022+0000 7f6a0a0d0700 1 --2- 172.21.15.78:0/476567884 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6a0406ab40 0x7f6a041ecfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:50.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.023+0000 7f6a098cf700 1 -- 172.21.15.78:0/476567884 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6a0406ab40 msgr2=0x7f6a041ecfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:50.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.023+0000 7f6a098cf700 1 --2- 172.21.15.78:0/476567884 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6a0406ab40 0x7f6a041ecfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:50.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.023+0000 7f6a098cf700 1 -- 172.21.15.78:0/476567884 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6a041554e0 msgr2=0x7f6a041577a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:50.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.023+0000 7f6a098cf700 1 --2- 172.21.15.78:0/476567884 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6a041554e0 0x7f6a041577a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:50.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.023+0000 7f6a098cf700 1 -- 172.21.15.78:0/476567884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69fc009b30 con 0x7f6a040c0350 2023-10-14T11:58:50.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.023+0000 7f6a0a0d0700 1 --2- 172.21.15.78:0/476567884 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6a0406ab40 0x7f6a041ecfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:50.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.023+0000 7f6a098cf700 1 --2- 172.21.15.78:0/476567884 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a040c0350 0x7f6a0414f990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6a0000b4a0 tx=0x7f6a0000b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:50.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.023+0000 7f69faffd700 1 -- 172.21.15.78:0/476567884 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a00011830 con 0x7f6a040c0350 2023-10-14T11:58:50.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.023+0000 7f69faffd700 1 -- 172.21.15.78:0/476567884 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a00011990 con 0x7f6a040c0350 2023-10-14T11:58:50.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.023+0000 7f6a0c334700 1 -- 172.21.15.78:0/476567884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a041edc60 con 0x7f6a040c0350 2023-10-14T11:58:50.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.023+0000 7f69faffd700 1 -- 172.21.15.78:0/476567884 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a00011d40 con 0x7f6a040c0350 2023-10-14T11:58:50.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.023+0000 7f6a0c334700 1 -- 172.21.15.78:0/476567884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a041ed4f0 con 0x7f6a040c0350 2023-10-14T11:58:50.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.024+0000 7f6a0c334700 1 -- 172.21.15.78:0/476567884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a040abd70 con 0x7f6a040c0350 2023-10-14T11:58:50.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.024+0000 7f69faffd700 1 -- 172.21.15.78:0/476567884 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6a0001a400 con 0x7f6a040c0350 2023-10-14T11:58:50.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.025+0000 7f69faffd700 1 --2- 172.21.15.78:0/476567884 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f69e806af00 0x7f69e806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:50.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.025+0000 7f6a0a0d0700 1 --2- 172.21.15.78:0/476567884 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f69e806af00 0x7f69e806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:50.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.025+0000 7f69faffd700 1 -- 172.21.15.78:0/476567884 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6a00089220 con 0x7f6a040c0350 2023-10-14T11:58:50.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.025+0000 7f6a0a0d0700 1 --2- 172.21.15.78:0/476567884 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f69e806af00 0x7f69e806d3c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f69f40099f0 tx=0x7f69f4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:50.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.029+0000 7f69faffd700 1 -- 172.21.15.78:0/476567884 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6a040abd70 con 0x7f6a040c0350 2023-10-14T11:58:50.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.259+0000 7f6a0c334700 1 -- 172.21.15.78:0/476567884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a040abd70 con 0x7f6a040c0350 2023-10-14T11:58:50.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.260+0000 7f69faffd700 1 -- 172.21.15.78:0/476567884 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4904 (secure 0 0 0) 0x7f6a040abd70 con 0x7f6a040c0350 2023-10-14T11:58:50.262 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:50.263 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/353043244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":353043244},{"type":"v1","addr":"172.21.15.78:6836","nonce":353043244}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6834/3389817343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":3389817343},{"type":"v1","addr":"172.21.15.195:6834","nonce":3389817343}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4267,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/460176701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6833","nonce":460176701},{"type":"v1","addr":"172.21.15.195:6835","nonce":460176701}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T11:58:14.535101+0000","modified":"2023-10-14T11:58:17.559202+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/920300787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":920300787},{"type":"v1","addr":"172.21.15.78:6837","nonce":920300787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-14T11:58:50.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.262+0000 7f6a0c334700 1 -- 172.21.15.78:0/476567884 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f69e806af00 msgr2=0x7f69e806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:50.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.262+0000 7f6a0c334700 1 --2- 172.21.15.78:0/476567884 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f69e806af00 0x7f69e806d3c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f69f40099f0 tx=0x7f69f4008040 comp rx=0 tx=0).stop 2023-10-14T11:58:50.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.262+0000 7f6a0c334700 1 -- 172.21.15.78:0/476567884 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a040c0350 msgr2=0x7f6a0414f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:50.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.262+0000 7f6a0c334700 1 --2- 172.21.15.78:0/476567884 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a040c0350 0x7f6a0414f990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6a0000b4a0 tx=0x7f6a0000b7b0 comp rx=0 tx=0).stop 2023-10-14T11:58:50.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.262+0000 7f6a0c334700 1 -- 172.21.15.78:0/476567884 shutdown_connections 2023-10-14T11:58:50.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.262+0000 7f6a0c334700 1 --2- 172.21.15.78:0/476567884 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f69e806af00 0x7f69e806d3c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:50.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.262+0000 7f6a0c334700 1 --2- 172.21.15.78:0/476567884 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6a041554e0 0x7f6a041577a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:50.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.262+0000 7f6a0c334700 1 --2- 172.21.15.78:0/476567884 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6a0406ab40 0x7f6a041ecfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:50.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.262+0000 7f6a0c334700 1 --2- 172.21.15.78:0/476567884 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a040c0350 0x7f6a0414f990 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:50.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.262+0000 7f6a0c334700 1 -- 172.21.15.78:0/476567884 >> 172.21.15.78:0/476567884 conn(0x7f6a040c3ac0 msgr2=0x7f6a0406af20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:50.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.262+0000 7f6a0c334700 1 -- 172.21.15.78:0/476567884 shutdown_connections 2023-10-14T11:58:50.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.262+0000 7f6a0c334700 1 -- 172.21.15.78:0/476567884 wait complete. 2023-10-14T11:58:50.268 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 8 2023-10-14T11:58:50.276 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T11:58:50.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.393+0000 7f25bec2f700 1 Processor -- start 2023-10-14T11:58:50.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.393+0000 7f25bec2f700 1 -- start start 2023-10-14T11:58:50.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.394+0000 7f25bec2f700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f25b806ab40 0x7f25b806f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:50.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.394+0000 7f25bec2f700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f25b81554e0 0x7f25b806af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:50.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.394+0000 7f25bec2f700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b80c49f0 con 0x7f25b80c0350 2023-10-14T11:58:50.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.394+0000 7f25bec2f700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25b80726a0 con 0x7f25b81554e0 2023-10-14T11:58:50.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.394+0000 7f25bec2f700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25b8072980 con 0x7f25b806ab40 2023-10-14T11:58:50.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.394+0000 7f25b7fff700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f25b81554e0 0x7f25b806af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:50.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.394+0000 7f25b7fff700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f25b81554e0 0x7f25b806af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45878/0 (socket says 172.21.15.78:45878) 2023-10-14T11:58:50.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.394+0000 7f25b7fff700 1 -- 172.21.15.78:0/1123923642 learned_addr learned my addr 172.21.15.78:0/1123923642 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:50.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.394+0000 7f25bc9cb700 1 --2- 172.21.15.78:0/1123923642 >> v2:172.21.15.195:3300/0 conn(0x7f25b806ab40 0x7f25b806f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:50.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.395+0000 7f25b7fff700 1 -- 172.21.15.78:0/1123923642 >> v2:172.21.15.195:3300/0 conn(0x7f25b806ab40 msgr2=0x7f25b806f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:50.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.395+0000 7f25b7fff700 1 --2- 172.21.15.78:0/1123923642 >> v2:172.21.15.195:3300/0 conn(0x7f25b806ab40 0x7f25b806f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:50.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.395+0000 7f25b7fff700 1 -- 172.21.15.78:0/1123923642 >> v1:172.21.15.195:6789/0 conn(0x7f25b80c0350 legacy=0x7f25b813d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:50.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.395+0000 7f25b7fff700 1 -- 172.21.15.78:0/1123923642 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b8154a00 con 0x7f25b81554e0 2023-10-14T11:58:50.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.395+0000 7f25b7fff700 1 --2- 172.21.15.78:0/1123923642 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25b81554e0 0x7f25b806af20 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f25b8072bc0 tx=0x7f25a4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0106ac92b68361 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:50.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.395+0000 7f25b77fe700 1 -- 172.21.15.78:0/1123923642 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a4011830 con 0x7f25b81554e0 2023-10-14T11:58:50.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.395+0000 7f25b77fe700 1 -- 172.21.15.78:0/1123923642 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25a4011990 con 0x7f25b81554e0 2023-10-14T11:58:50.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.395+0000 7f25b77fe700 1 -- 172.21.15.78:0/1123923642 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a4011d20 con 0x7f25b81554e0 2023-10-14T11:58:50.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.395+0000 7f25bec2f700 1 -- 172.21.15.78:0/1123923642 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25b81554e0 msgr2=0x7f25b806af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:50.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.395+0000 7f25bec2f700 1 --2- 172.21.15.78:0/1123923642 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25b81554e0 0x7f25b806af20 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f25b8072bc0 tx=0x7f25a4009bd0 comp rx=0 tx=0).stop 2023-10-14T11:58:50.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.396+0000 7f25bec2f700 1 -- 172.21.15.78:0/1123923642 shutdown_connections 2023-10-14T11:58:50.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.396+0000 7f25bec2f700 1 --2- 172.21.15.78:0/1123923642 >> v2:172.21.15.195:3300/0 conn(0x7f25b806ab40 0x7f25b806f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:50.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.396+0000 7f25bec2f700 1 --2- 172.21.15.78:0/1123923642 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25b81554e0 0x7f25b806af20 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:50.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.396+0000 7f25bec2f700 1 -- 172.21.15.78:0/1123923642 >> 172.21.15.78:0/1123923642 conn(0x7f25b80c3ac0 msgr2=0x7f25b80c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:50.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.396+0000 7f25bec2f700 1 -- 172.21.15.78:0/1123923642 shutdown_connections 2023-10-14T11:58:50.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.396+0000 7f25bec2f700 1 -- 172.21.15.78:0/1123923642 wait complete. 2023-10-14T11:58:50.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.397+0000 7f25bec2f700 1 Processor -- start 2023-10-14T11:58:50.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.397+0000 7f25bec2f700 1 -- start start 2023-10-14T11:58:50.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.397+0000 7f25bec2f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25b806ab40 0x7f25b8151280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:50.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.397+0000 7f25bec2f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f25b80c0350 0x7f25b8138980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:50.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.398+0000 7f25bec2f700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f25b81554e0 0x7f25b8157600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:50.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.398+0000 7f25bec2f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b8158250 con 0x7f25b806ab40 2023-10-14T11:58:50.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.398+0000 7f25bec2f700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b81ed710 con 0x7f25b80c0350 2023-10-14T11:58:50.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.398+0000 7f25bec2f700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25b814fc40 con 0x7f25b81554e0 2023-10-14T11:58:50.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.398+0000 7f25bc9cb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25b806ab40 0x7f25b8151280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:50.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.398+0000 7f25bc9cb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25b806ab40 0x7f25b8151280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45884/0 (socket says 172.21.15.78:45884) 2023-10-14T11:58:50.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.398+0000 7f25bc9cb700 1 -- 172.21.15.78:0/2271072768 learned_addr learned my addr 172.21.15.78:0/2271072768 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:50.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.398+0000 7f25b7fff700 1 --2- 172.21.15.78:0/2271072768 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f25b80c0350 0x7f25b8138980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:50.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.398+0000 7f25bd1cc700 1 --2- 172.21.15.78:0/2271072768 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f25b81554e0 0x7f25b8157600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:50.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.398+0000 7f25bc9cb700 1 -- 172.21.15.78:0/2271072768 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f25b81554e0 msgr2=0x7f25b8157600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:50.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.398+0000 7f25bc9cb700 1 --2- 172.21.15.78:0/2271072768 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f25b81554e0 0x7f25b8157600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:50.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.398+0000 7f25bc9cb700 1 -- 172.21.15.78:0/2271072768 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f25b80c0350 msgr2=0x7f25b8138980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:50.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.398+0000 7f25bc9cb700 1 --2- 172.21.15.78:0/2271072768 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f25b80c0350 0x7f25b8138980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:50.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.398+0000 7f25bc9cb700 1 -- 172.21.15.78:0/2271072768 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a4009550 con 0x7f25b806ab40 2023-10-14T11:58:50.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.398+0000 7f25bc9cb700 1 --2- 172.21.15.78:0/2271072768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25b806ab40 0x7f25b8151280 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f25ac00e9d0 tx=0x7f25ac00ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:50.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.398+0000 7f25b57fa700 1 -- 172.21.15.78:0/2271072768 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ac009730 con 0x7f25b806ab40 2023-10-14T11:58:50.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.398+0000 7f25b57fa700 1 -- 172.21.15.78:0/2271072768 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25ac00c850 con 0x7f25b806ab40 2023-10-14T11:58:50.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.398+0000 7f25bec2f700 1 -- 172.21.15.78:0/2271072768 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25b81ecb00 con 0x7f25b806ab40 2023-10-14T11:58:50.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.398+0000 7f25b57fa700 1 -- 172.21.15.78:0/2271072768 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ac00cbe0 con 0x7f25b806ab40 2023-10-14T11:58:50.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.398+0000 7f25bec2f700 1 -- 172.21.15.78:0/2271072768 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25b81ed0c0 con 0x7f25b806ab40 2023-10-14T11:58:50.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.399+0000 7f25bec2f700 1 -- 172.21.15.78:0/2271072768 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25b81ed880 con 0x7f25b806ab40 2023-10-14T11:58:50.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.400+0000 7f25b57fa700 1 -- 172.21.15.78:0/2271072768 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f25ac00cd40 con 0x7f25b806ab40 2023-10-14T11:58:50.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.400+0000 7f25b57fa700 1 --2- 172.21.15.78:0/2271072768 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f259c06af00 0x7f259c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:50.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.400+0000 7f25b7fff700 1 --2- 172.21.15.78:0/2271072768 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f259c06af00 0x7f259c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:50.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.400+0000 7f25b57fa700 1 -- 172.21.15.78:0/2271072768 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f25ac089690 con 0x7f25b806ab40 2023-10-14T11:58:50.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.400+0000 7f25b7fff700 1 --2- 172.21.15.78:0/2271072768 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f259c06af00 0x7f259c06d3c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f25a4014040 tx=0x7f25a4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:50.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.405+0000 7f25b57fa700 1 -- 172.21.15.78:0/2271072768 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f25ac0547d0 con 0x7f25b806ab40 2023-10-14T11:58:50.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.595+0000 7f25bec2f700 1 -- 172.21.15.78:0/2271072768 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25b8074b80 con 0x7f25b806ab40 2023-10-14T11:58:50.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.596+0000 7f25b57fa700 1 -- 172.21.15.78:0/2271072768 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16481 (secure 0 0 0) 0x7f25ac057df0 con 0x7f25b806ab40 2023-10-14T11:58:50.599 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:50.599 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":21,"fsid":"4c5dc134-3092-406a-a1e1-13e6d9e926e8","created":"2023-10-14T11:57:57.363750+0000","modified":"2023-10-14T11:58:16.139413+0000","last_up_change":"2023-10-14T11:58:06.982472+0000","last_in_change":"2023-10-14T11:57:59.821373+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T11:58:09.001200+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T11:58:12.778573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T11:58:13.490286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"625a5fc0-9474-4eae-b80e-14c27a74ad19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6801","nonce":3884969780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6803","nonce":3884969780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6807","nonce":3884969780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6805","nonce":3884969780}]},"public_addr":"172.21.15.78:6801/3884969780","cluster_addr":"172.21.15.78:6803/3884969780","heartbeat_back_addr":"172.21.15.78:6807/3884969780","heartbeat_front_addr":"172.21.15.78:6805/3884969780","state":["exists","up"]},{"osd":1,"uuid":"49d63db1-2981-49ab-9c40-c50fc744b982","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6809","nonce":2176928406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6811","nonce":2176928406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6815","nonce":2176928406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6813","nonce":2176928406}]},"public_addr":"172.21.15.78:6809/2176928406","cluster_addr":"172.21.15.78:6811/2176928406","heartbeat_back_addr":"172.21.15.78:6815/2176928406","heartbeat_front_addr":"172.21.15.78:6813/2176928406","state":["exists","up"]},{"osd":2,"uuid":"1d7d0b29-cbd1-4717-9fe6-765a8d4998b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6825","nonce":3647044577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6827","nonce":3647044577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6831","nonce":3647044577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6829","nonce":3647044577}]},"public_addr":"172.21.15.78:6825/3647044577","cluster_addr":"172.21.15.78:6827/3647044577","heartbeat_back_addr":"172.21.15.78:6831/3647044577","heartbeat_front_addr":"172.21.15.78:6829/3647044577","state":["exists","up"]},{"osd":3,"uuid":"2f75ded1-e9e2-40e7-99ae-45e1e425e1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6817","nonce":3584112288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6819","nonce":3584112288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6823","nonce":3584112288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6821","nonce":3584112288}]},"public_addr":"172.21.15.78:6817/3584112288","cluster_addr":"172.21.15.78:6819/3584112288","heartbeat_back_addr":"172.21.15.78:6823/3584112288","heartbeat_front_addr":"172.21.15.78:6821/3584112288","state":["exists","up"]},{"osd":4,"uuid":"5130b432-8358-46cb-8a31-3064c976f6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6801","nonce":3732756301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6803","nonce":3732756301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6807","nonce":3732756301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6805","nonce":3732756301}]},"public_addr":"172.21.15.195:6801/3732756301","cluster_addr":"172.21.15.195:6803/3732756301","heartbeat_back_addr":"172.21.15.195:6807/3732756301","heartbeat_front_addr":"172.21.15.195:6805/3732756301","state":["exists","up"]},{"osd":5,"uuid":"60e234a9-a506-4e40-9c19-49a58f3cf28c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6825","nonce":2857706411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6827","nonce":2857706411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6831","nonce":2857706411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6829","nonce":2857706411}]},"public_addr":"172.21.15.195:6825/2857706411","cluster_addr":"172.21.15.195:6827/2857706411","heartbeat_back_addr":"172.21.15.195:6831/2857706411","heartbeat_front_addr":"172.21.15.195:6829/2857706411","state":["exists","up"]},{"osd":6,"uuid":"df7d2dd6-26b6-4eb1-96e5-94d1f3955b33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6810","nonce":72554374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6813","nonce":72554374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6820","nonce":72554374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6817","nonce":72554374}]},"public_addr":"172.21.15.195:6810/72554374","cluster_addr":"172.21.15.195:6813/72554374","heartbeat_back_addr":"172.21.15.195:6820/72554374","heartbeat_front_addr":"172.21.15.195:6817/72554374","state":["exists","up"]},{"osd":7,"uuid":"47a9d4de-f56d-4aa4-9900-c4a9cf9b2f97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6809","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6811","nonce":482472979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6816","nonce":482472979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6823","nonce":482472979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6821","nonce":482472979}]},"public_addr":"172.21.15.195:6811/482472979","cluster_addr":"172.21.15.195:6816/482472979","heartbeat_back_addr":"172.21.15.195:6823/482472979","heartbeat_front_addr":"172.21.15.195:6821/482472979","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.754003+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.005994+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.043603+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.024506+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.062585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.033767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.994825+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.038300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T11:58:50.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.599+0000 7f25bec2f700 1 -- 172.21.15.78:0/2271072768 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f259c06af00 msgr2=0x7f259c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:50.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.599+0000 7f25bec2f700 1 --2- 172.21.15.78:0/2271072768 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f259c06af00 0x7f259c06d3c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f25a4014040 tx=0x7f25a4008030 comp rx=0 tx=0).stop 2023-10-14T11:58:50.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.599+0000 7f25bec2f700 1 -- 172.21.15.78:0/2271072768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25b806ab40 msgr2=0x7f25b8151280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:50.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.599+0000 7f25bec2f700 1 --2- 172.21.15.78:0/2271072768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25b806ab40 0x7f25b8151280 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f25ac00e9d0 tx=0x7f25ac00ed90 comp rx=0 tx=0).stop 2023-10-14T11:58:50.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.599+0000 7f25bec2f700 1 -- 172.21.15.78:0/2271072768 shutdown_connections 2023-10-14T11:58:50.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.599+0000 7f25bec2f700 1 --2- 172.21.15.78:0/2271072768 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f25b81554e0 0x7f25b8157600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:50.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.599+0000 7f25bec2f700 1 --2- 172.21.15.78:0/2271072768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25b806ab40 0x7f25b8151280 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:50.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.599+0000 7f25bec2f700 1 --2- 172.21.15.78:0/2271072768 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f259c06af00 0x7f259c06d3c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:50.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.599+0000 7f25bec2f700 1 --2- 172.21.15.78:0/2271072768 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f25b80c0350 0x7f25b8138980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:50.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.599+0000 7f25bec2f700 1 -- 172.21.15.78:0/2271072768 >> 172.21.15.78:0/2271072768 conn(0x7f25b80c3ac0 msgr2=0x7f25b80c0750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:50.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.599+0000 7f25bec2f700 1 -- 172.21.15.78:0/2271072768 shutdown_connections 2023-10-14T11:58:50.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.599+0000 7f25bec2f700 1 -- 172.21.15.78:0/2271072768 wait complete. 2023-10-14T11:58:50.614 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-14T11:58:50.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.733+0000 7efc1c5e2700 1 Processor -- start 2023-10-14T11:58:50.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.733+0000 7efc1c5e2700 1 -- start start 2023-10-14T11:58:50.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.734+0000 7efc1c5e2700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7efc140c4540 0x7efc140c02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:50.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.734+0000 7efc1c5e2700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc1406ebe0 con 0x7efc1406b500 2023-10-14T11:58:50.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.734+0000 7efc1c5e2700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc141558c0 con 0x7efc1413dbe0 2023-10-14T11:58:50.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.734+0000 7efc1c5e2700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc1406d330 con 0x7efc140c4540 2023-10-14T11:58:50.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.734+0000 7efc19b7d700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7efc140c4540 0x7efc140c02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:50.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.734+0000 7efc1ab7f700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7efc1406b500 0x7efc140c0820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50772/0 (socket says 172.21.15.78:50772) 2023-10-14T11:58:50.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.734+0000 7efc1ab7f700 1 -- 172.21.15.78:0/3303869357 learned_addr learned my addr 172.21.15.78:0/3303869357 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:50.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.734+0000 7efc19b7d700 1 -- 172.21.15.78:0/3303869357 >> v1:172.21.15.195:6789/0 conn(0x7efc1413dbe0 legacy=0x7efc1406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:50.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.734+0000 7efc19b7d700 1 -- 172.21.15.78:0/3303869357 >> v1:172.21.15.78:6789/0 conn(0x7efc1406b500 legacy=0x7efc140c0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:50.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.734+0000 7efc1937c700 1 -- 172.21.15.78:0/3303869357 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2551746353 0 0) 0x7efc1406ebe0 con 0x7efc1406b500 2023-10-14T11:58:50.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.734+0000 7efc19b7d700 1 -- 172.21.15.78:0/3303869357 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc14154b00 con 0x7efc140c4540 2023-10-14T11:58:50.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.735+0000 7efc19b7d700 1 --2- 172.21.15.78:0/3303869357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc140c4540 0x7efc140c02e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7efc10009840 tx=0x7efc10009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b24d26ab05197283 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:50.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.735+0000 7efc1937c700 1 -- 172.21.15.78:0/3303869357 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc10004230 con 0x7efc140c4540 2023-10-14T11:58:50.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.735+0000 7efc1937c700 1 -- 172.21.15.78:0/3303869357 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc10004390 con 0x7efc140c4540 2023-10-14T11:58:50.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.735+0000 7efc1937c700 1 -- 172.21.15.78:0/3303869357 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc100036a0 con 0x7efc140c4540 2023-10-14T11:58:50.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.735+0000 7efc1c5e2700 1 -- 172.21.15.78:0/3303869357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc140c4540 msgr2=0x7efc140c02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:50.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.735+0000 7efc1c5e2700 1 --2- 172.21.15.78:0/3303869357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc140c4540 0x7efc140c02e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7efc10009840 tx=0x7efc10009b50 comp rx=0 tx=0).stop 2023-10-14T11:58:50.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.735+0000 7efc1c5e2700 1 -- 172.21.15.78:0/3303869357 shutdown_connections 2023-10-14T11:58:50.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.735+0000 7efc1c5e2700 1 --2- 172.21.15.78:0/3303869357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc140c4540 0x7efc140c02e0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:50.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.735+0000 7efc1c5e2700 1 -- 172.21.15.78:0/3303869357 >> 172.21.15.78:0/3303869357 conn(0x7efc140c3a50 msgr2=0x7efc140c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:50.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.735+0000 7efc1c5e2700 1 -- 172.21.15.78:0/3303869357 shutdown_connections 2023-10-14T11:58:50.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.735+0000 7efc1c5e2700 1 -- 172.21.15.78:0/3303869357 wait complete. 2023-10-14T11:58:50.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.736+0000 7efc1c5e2700 1 Processor -- start 2023-10-14T11:58:50.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.736+0000 7efc1c5e2700 1 -- start start 2023-10-14T11:58:50.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.737+0000 7efc1c5e2700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efc1406b500 0x7efc141368f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:50.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.737+0000 7efc1c5e2700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efc140c4540 0x7efc141ee4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:50.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.737+0000 7efc1c5e2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc1413dbe0 0x7efc140c5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:50.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.737+0000 7efc1c5e2700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc14157fe0 con 0x7efc1413dbe0 2023-10-14T11:58:50.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.737+0000 7efc1c5e2700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc140ac460 con 0x7efc140c4540 2023-10-14T11:58:50.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.737+0000 7efc1c5e2700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc140abea0 con 0x7efc1406b500 2023-10-14T11:58:50.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.737+0000 7efc1ab7f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc1413dbe0 0x7efc140c5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:50.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.737+0000 7efc1ab7f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc1413dbe0 0x7efc140c5e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45902/0 (socket says 172.21.15.78:45902) 2023-10-14T11:58:50.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.737+0000 7efc1ab7f700 1 -- 172.21.15.78:0/2379629443 learned_addr learned my addr 172.21.15.78:0/2379629443 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:50.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.737+0000 7efc1a37e700 1 --2- 172.21.15.78:0/2379629443 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efc1406b500 0x7efc141368f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:50.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.737+0000 7efc19b7d700 1 --2- 172.21.15.78:0/2379629443 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efc140c4540 0x7efc141ee4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:50.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.738+0000 7efc1ab7f700 1 -- 172.21.15.78:0/2379629443 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efc1406b500 msgr2=0x7efc141368f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:50.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.738+0000 7efc1ab7f700 1 --2- 172.21.15.78:0/2379629443 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efc1406b500 0x7efc141368f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:50.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.738+0000 7efc1ab7f700 1 -- 172.21.15.78:0/2379629443 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efc140c4540 msgr2=0x7efc141ee4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:50.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.738+0000 7efc1ab7f700 1 --2- 172.21.15.78:0/2379629443 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efc140c4540 0x7efc141ee4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:50.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.738+0000 7efc1ab7f700 1 -- 172.21.15.78:0/2379629443 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc100094a0 con 0x7efc1413dbe0 2023-10-14T11:58:50.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.738+0000 7efc1a37e700 1 --2- 172.21.15.78:0/2379629443 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efc1406b500 0x7efc141368f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:50.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.738+0000 7efc1ab7f700 1 --2- 172.21.15.78:0/2379629443 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc1413dbe0 0x7efc140c5e20 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7efc0c00ca80 tx=0x7efc0c00ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:50.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.738+0000 7efc0affd700 1 -- 172.21.15.78:0/2379629443 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0c00aaa0 con 0x7efc1413dbe0 2023-10-14T11:58:50.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.738+0000 7efc0affd700 1 -- 172.21.15.78:0/2379629443 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc0c00ac00 con 0x7efc1413dbe0 2023-10-14T11:58:50.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.738+0000 7efc1c5e2700 1 -- 172.21.15.78:0/2379629443 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc140ac180 con 0x7efc1413dbe0 2023-10-14T11:58:50.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.738+0000 7efc0affd700 1 -- 172.21.15.78:0/2379629443 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0c0070d0 con 0x7efc1413dbe0 2023-10-14T11:58:50.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.738+0000 7efc1c5e2700 1 -- 172.21.15.78:0/2379629443 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc141edfa0 con 0x7efc1413dbe0 2023-10-14T11:58:50.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.739+0000 7efc1c5e2700 1 -- 172.21.15.78:0/2379629443 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc14074b80 con 0x7efc1413dbe0 2023-10-14T11:58:50.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.739+0000 7efc0affd700 1 -- 172.21.15.78:0/2379629443 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7efc0c007230 con 0x7efc1413dbe0 2023-10-14T11:58:50.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.740+0000 7efc0affd700 1 --2- 172.21.15.78:0/2379629443 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7efbf806af00 0x7efbf806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:50.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.740+0000 7efc0affd700 1 -- 172.21.15.78:0/2379629443 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efc0c089290 con 0x7efc1413dbe0 2023-10-14T11:58:50.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.740+0000 7efc1a37e700 1 --2- 172.21.15.78:0/2379629443 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7efbf806af00 0x7efbf806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:50.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.740+0000 7efc1a37e700 1 --2- 172.21.15.78:0/2379629443 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7efbf806af00 0x7efbf806d3c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7efc04003fc0 tx=0x7efc04009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:50.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.744+0000 7efc0affd700 1 -- 172.21.15.78:0/2379629443 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7efc0c0543d0 con 0x7efc1413dbe0 2023-10-14T11:58:50.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.975+0000 7efc1c5e2700 1 -- 172.21.15.78:0/2379629443 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7efc141ee190 con 0x7efc1413dbe0 2023-10-14T11:58:50.977 INFO:tasks.ceph.mon.a.smithi078.stderr:2023-10-14T11:58:50.975+0000 7f956c6f2700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-14T11:58:50.978 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.977+0000 7efc0affd700 1 -- 172.21.15.78:0/2379629443 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7efc0c0579f0 con 0x7efc1413dbe0 2023-10-14T11:58:50.982 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.980+0000 7efc1c5e2700 1 -- 172.21.15.78:0/2379629443 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7efbf806af00 msgr2=0x7efbf806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:50.982 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.980+0000 7efc1c5e2700 1 --2- 172.21.15.78:0/2379629443 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7efbf806af00 0x7efbf806d3c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7efc04003fc0 tx=0x7efc04009040 comp rx=0 tx=0).stop 2023-10-14T11:58:50.982 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.980+0000 7efc1c5e2700 1 -- 172.21.15.78:0/2379629443 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc1413dbe0 msgr2=0x7efc140c5e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:50.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.980+0000 7efc1c5e2700 1 --2- 172.21.15.78:0/2379629443 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc1413dbe0 0x7efc140c5e20 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7efc0c00ca80 tx=0x7efc0c00ce40 comp rx=0 tx=0).stop 2023-10-14T11:58:50.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.980+0000 7efc1c5e2700 1 -- 172.21.15.78:0/2379629443 shutdown_connections 2023-10-14T11:58:50.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.980+0000 7efc1c5e2700 1 --2- 172.21.15.78:0/2379629443 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc1413dbe0 0x7efc140c5e20 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:50.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.980+0000 7efc1c5e2700 1 --2- 172.21.15.78:0/2379629443 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efc1406b500 0x7efc141368f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:50.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.980+0000 7efc1c5e2700 1 --2- 172.21.15.78:0/2379629443 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7efbf806af00 0x7efbf806d3c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:50.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.980+0000 7efc1c5e2700 1 --2- 172.21.15.78:0/2379629443 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efc140c4540 0x7efc141ee4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:50.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.980+0000 7efc1c5e2700 1 -- 172.21.15.78:0/2379629443 >> 172.21.15.78:0/2379629443 conn(0x7efc140c3a50 msgr2=0x7efc140c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:50.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.980+0000 7efc1c5e2700 1 -- 172.21.15.78:0/2379629443 shutdown_connections 2023-10-14T11:58:50.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:50.980+0000 7efc1c5e2700 1 -- 172.21.15.78:0/2379629443 wait complete. 2023-10-14T11:58:50.985 INFO:teuthology.orchestra.run.smithi078.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-14T11:58:50.994 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-14T11:58:51.018 INFO:tasks.ceph.mds.a.smithi078.stdout:starting mds.a at 2023-10-14T11:58:51.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.115+0000 7fc33a385700 1 Processor -- start 2023-10-14T11:58:51.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.115+0000 7fc33a385700 1 -- start start 2023-10-14T11:58:51.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.116+0000 7fc33a385700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fc3340c35e0 0x7fc3341535d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:51.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.116+0000 7fc33a385700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3340c26d0 con 0x7fc334154c20 2023-10-14T11:58:51.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.116+0000 7fc33a385700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3340c29b0 con 0x7fc33414ed10 2023-10-14T11:58:51.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.116+0000 7fc33a385700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3340a8520 con 0x7fc3340c35e0 2023-10-14T11:58:51.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.116+0000 7fc338922700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc334154c20 0x7fc334155030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50798/0 (socket says 172.21.15.78:50798) 2023-10-14T11:58:51.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.116+0000 7fc338922700 1 -- 172.21.15.78:0/2703619921 learned_addr learned my addr 172.21.15.78:0/2703619921 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:51.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.116+0000 7fc3337fe700 1 --2- 172.21.15.78:0/2703619921 >> v2:172.21.15.195:3300/0 conn(0x7fc3340c35e0 0x7fc3341535d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:51.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.116+0000 7fc332ffd700 1 -- 172.21.15.78:0/2703619921 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 813058933 0 0) 0x7fc3340c26d0 con 0x7fc334154c20 2023-10-14T11:58:51.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.116+0000 7fc332ffd700 1 -- 172.21.15.78:0/2703619921 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc31c0057c0 con 0x7fc334154c20 2023-10-14T11:58:51.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.117+0000 7fc332ffd700 1 -- 172.21.15.78:0/2703619921 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 744795803 0 0) 0x7fc31c0057c0 con 0x7fc334154c20 2023-10-14T11:58:51.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.117+0000 7fc332ffd700 1 -- 172.21.15.78:0/2703619921 >> v2:172.21.15.195:3300/0 conn(0x7fc3340c35e0 msgr2=0x7fc3341535d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:51.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.117+0000 7fc332ffd700 1 --2- 172.21.15.78:0/2703619921 >> v2:172.21.15.195:3300/0 conn(0x7fc3340c35e0 0x7fc3341535d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:51.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.117+0000 7fc332ffd700 1 -- 172.21.15.78:0/2703619921 >> v1:172.21.15.195:6789/0 conn(0x7fc33414ed10 legacy=0x7fc334152ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:51.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.117+0000 7fc332ffd700 1 -- 172.21.15.78:0/2703619921 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3340c2c90 con 0x7fc334154c20 2023-10-14T11:58:51.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.117+0000 7fc332ffd700 1 -- 172.21.15.78:0/2703619921 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441799917 0 0) 0x7fc32c0019f0 con 0x7fc334154c20 2023-10-14T11:58:51.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.117+0000 7fc332ffd700 1 -- 172.21.15.78:0/2703619921 >> v1:172.21.15.78:6789/0 conn(0x7fc334154c20 legacy=0x7fc334155030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:51.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.117+0000 7fc332ffd700 1 --2- 172.21.15.78:0/2703619921 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc31c005c10 0x7fc31c006030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:51.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.117+0000 7fc332ffd700 1 --2- 172.21.15.78:0/2703619921 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc31c006a60 0x7fc31c006ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:51.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.117+0000 7fc332ffd700 1 --2- 172.21.15.78:0/2703619921 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc31c007420 0x7fc31c00dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:51.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.117+0000 7fc332ffd700 1 -- 172.21.15.78:0/2703619921 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3340a8520 con 0x7fc31c006a60 2023-10-14T11:58:51.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.117+0000 7fc332ffd700 1 -- 172.21.15.78:0/2703619921 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc31c00e3c0 con 0x7fc31c005c10 2023-10-14T11:58:51.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.117+0000 7fc332ffd700 1 -- 172.21.15.78:0/2703619921 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc31c00e580 con 0x7fc31c007420 2023-10-14T11:58:51.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.117+0000 7fc3337fe700 1 --2- 172.21.15.78:0/2703619921 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc31c006a60 0x7fc31c006ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:51.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.117+0000 7fc333fff700 1 --2- 172.21.15.78:0/2703619921 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc31c005c10 0x7fc31c006030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:51.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.117+0000 7fc338922700 1 --2- 172.21.15.78:0/2703619921 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc31c007420 0x7fc31c00dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:51.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.117+0000 7fc3337fe700 1 -- 172.21.15.78:0/2703619921 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc31c007420 msgr2=0x7fc31c00dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:51.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.117+0000 7fc3337fe700 1 --2- 172.21.15.78:0/2703619921 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc31c007420 0x7fc31c00dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:51.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.117+0000 7fc3337fe700 1 -- 172.21.15.78:0/2703619921 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc31c005c10 msgr2=0x7fc31c006030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:51.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.117+0000 7fc3337fe700 1 --2- 172.21.15.78:0/2703619921 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc31c005c10 0x7fc31c006030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:51.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.117+0000 7fc3337fe700 1 -- 172.21.15.78:0/2703619921 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc31c0057c0 con 0x7fc31c006a60 2023-10-14T11:58:51.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.117+0000 7fc338922700 1 --2- 172.21.15.78:0/2703619921 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc31c007420 0x7fc31c00dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:51.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.117+0000 7fc338922700 1 -- 172.21.15.78:0/2703619921 reap_dead start 2023-10-14T11:58:51.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.117+0000 7fc3337fe700 1 --2- 172.21.15.78:0/2703619921 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc31c006a60 0x7fc31c006ee0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc324009b60 tx=0x7fc31c007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ebc77ed13d9f6ce server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:51.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.117+0000 7fc332ffd700 1 -- 172.21.15.78:0/2703619921 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32400e6c0 con 0x7fc31c006a60 2023-10-14T11:58:51.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.117+0000 7fc332ffd700 1 -- 172.21.15.78:0/2703619921 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc32400e820 con 0x7fc31c006a60 2023-10-14T11:58:51.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.117+0000 7fc332ffd700 1 -- 172.21.15.78:0/2703619921 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32400eb90 con 0x7fc31c006a60 2023-10-14T11:58:51.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.118+0000 7fc33a385700 1 -- 172.21.15.78:0/2703619921 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc31c006a60 msgr2=0x7fc31c006ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:51.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.118+0000 7fc33a385700 1 --2- 172.21.15.78:0/2703619921 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc31c006a60 0x7fc31c006ee0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc324009b60 tx=0x7fc31c007a90 comp rx=0 tx=0).stop 2023-10-14T11:58:51.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.118+0000 7fc33a385700 1 -- 172.21.15.78:0/2703619921 shutdown_connections 2023-10-14T11:58:51.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.118+0000 7fc33a385700 1 --2- 172.21.15.78:0/2703619921 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc31c006a60 0x7fc31c006ee0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:51.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.118+0000 7fc33a385700 1 -- 172.21.15.78:0/2703619921 >> 172.21.15.78:0/2703619921 conn(0x7fc3340c3a50 msgr2=0x7fc3340c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:51.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.118+0000 7fc33a385700 1 -- 172.21.15.78:0/2703619921 shutdown_connections 2023-10-14T11:58:51.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.118+0000 7fc33a385700 1 -- 172.21.15.78:0/2703619921 wait complete. 2023-10-14T11:58:51.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.119+0000 7fc33a385700 1 Processor -- start 2023-10-14T11:58:51.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.120+0000 7fc33a385700 1 -- start start 2023-10-14T11:58:51.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.120+0000 7fc33a385700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc31c006a60 0x7fc33406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:51.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.120+0000 7fc33a385700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3340ab3b0 0x7fc334151b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:51.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.120+0000 7fc33a385700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc3341520d0 0x7fc3340aa7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:51.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.121+0000 7fc33a385700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33406f620 con 0x7fc31c006a60 2023-10-14T11:58:51.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.121+0000 7fc33a385700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3341505a0 con 0x7fc3340ab3b0 2023-10-14T11:58:51.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.121+0000 7fc33a385700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc33414fb60 con 0x7fc3341520d0 2023-10-14T11:58:51.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.121+0000 7fc333fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc31c006a60 0x7fc33406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:51.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.121+0000 7fc333fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc31c006a60 0x7fc33406f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45930/0 (socket says 172.21.15.78:45930) 2023-10-14T11:58:51.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.121+0000 7fc333fff700 1 -- 172.21.15.78:0/1979208488 learned_addr learned my addr 172.21.15.78:0/1979208488 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:51.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.121+0000 7fc3337fe700 1 --2- 172.21.15.78:0/1979208488 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3340ab3b0 0x7fc334151b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:51.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.121+0000 7fc338922700 1 --2- 172.21.15.78:0/1979208488 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc3341520d0 0x7fc3340aa7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:51.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.121+0000 7fc333fff700 1 -- 172.21.15.78:0/1979208488 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc3341520d0 msgr2=0x7fc3340aa7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:51.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.121+0000 7fc333fff700 1 --2- 172.21.15.78:0/1979208488 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc3341520d0 0x7fc3340aa7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:51.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.121+0000 7fc333fff700 1 -- 172.21.15.78:0/1979208488 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3340ab3b0 msgr2=0x7fc334151b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:51.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.121+0000 7fc333fff700 1 --2- 172.21.15.78:0/1979208488 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3340ab3b0 0x7fc334151b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:51.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.121+0000 7fc333fff700 1 -- 172.21.15.78:0/1979208488 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc33414f880 con 0x7fc31c006a60 2023-10-14T11:58:51.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.121+0000 7fc3337fe700 1 --2- 172.21.15.78:0/1979208488 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3340ab3b0 0x7fc334151b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:51.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.121+0000 7fc333fff700 1 --2- 172.21.15.78:0/1979208488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc31c006a60 0x7fc33406f0e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc32800f820 tx=0x7fc32800fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:51.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.121+0000 7fc330ff9700 1 -- 172.21.15.78:0/1979208488 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3280052e0 con 0x7fc31c006a60 2023-10-14T11:58:51.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.121+0000 7fc330ff9700 1 -- 172.21.15.78:0/1979208488 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc328005440 con 0x7fc31c006a60 2023-10-14T11:58:51.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.121+0000 7fc33a385700 1 -- 172.21.15.78:0/1979208488 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc33414ff50 con 0x7fc31c006a60 2023-10-14T11:58:51.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.121+0000 7fc330ff9700 1 -- 172.21.15.78:0/1979208488 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3280038c0 con 0x7fc31c006a60 2023-10-14T11:58:51.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.121+0000 7fc33a385700 1 -- 172.21.15.78:0/1979208488 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3340728c0 con 0x7fc31c006a60 2023-10-14T11:58:51.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.122+0000 7fc33a385700 1 -- 172.21.15.78:0/1979208488 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3341524b0 con 0x7fc31c006a60 2023-10-14T11:58:51.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.123+0000 7fc330ff9700 1 -- 172.21.15.78:0/1979208488 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc328003a20 con 0x7fc31c006a60 2023-10-14T11:58:51.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.123+0000 7fc330ff9700 1 --2- 172.21.15.78:0/1979208488 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc32006af00 0x7fc32006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:51.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.123+0000 7fc330ff9700 1 -- 172.21.15.78:0/1979208488 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc3280893c0 con 0x7fc31c006a60 2023-10-14T11:58:51.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.123+0000 7fc3337fe700 1 --2- 172.21.15.78:0/1979208488 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc32006af00 0x7fc32006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:51.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.123+0000 7fc3337fe700 1 --2- 172.21.15.78:0/1979208488 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc32006af00 0x7fc32006d3c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc334153c40 tx=0x7fc324008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:51.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.127+0000 7fc330ff9700 1 -- 172.21.15.78:0/1979208488 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc3280544a0 con 0x7fc31c006a60 2023-10-14T11:58:51.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.365+0000 7fc33a385700 1 -- 172.21.15.78:0/1979208488 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc3340aa600 con 0x7fc31c006a60 2023-10-14T11:58:51.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.979+0000 7fc330ff9700 1 -- 172.21.15.78:0/1979208488 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc328057ac0 con 0x7fc31c006a60 2023-10-14T11:58:51.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.981+0000 7fc33a385700 1 -- 172.21.15.78:0/1979208488 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc32006af00 msgr2=0x7fc32006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:51.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.981+0000 7fc33a385700 1 --2- 172.21.15.78:0/1979208488 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc32006af00 0x7fc32006d3c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc334153c40 tx=0x7fc324008030 comp rx=0 tx=0).stop 2023-10-14T11:58:51.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.981+0000 7fc33a385700 1 -- 172.21.15.78:0/1979208488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc31c006a60 msgr2=0x7fc33406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:51.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.981+0000 7fc33a385700 1 --2- 172.21.15.78:0/1979208488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc31c006a60 0x7fc33406f0e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc32800f820 tx=0x7fc32800fbe0 comp rx=0 tx=0).stop 2023-10-14T11:58:51.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.981+0000 7fc33a385700 1 -- 172.21.15.78:0/1979208488 shutdown_connections 2023-10-14T11:58:51.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.981+0000 7fc33a385700 1 --2- 172.21.15.78:0/1979208488 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc3341520d0 0x7fc3340aa7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:51.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.981+0000 7fc33a385700 1 --2- 172.21.15.78:0/1979208488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc31c006a60 0x7fc33406f0e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:51.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.981+0000 7fc33a385700 1 --2- 172.21.15.78:0/1979208488 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc32006af00 0x7fc32006d3c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:51.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.981+0000 7fc33a385700 1 --2- 172.21.15.78:0/1979208488 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3340ab3b0 0x7fc334151b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:51.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.981+0000 7fc33a385700 1 -- 172.21.15.78:0/1979208488 >> 172.21.15.78:0/1979208488 conn(0x7fc3340c3a50 msgr2=0x7fc3340c1850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:51.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.981+0000 7fc33a385700 1 -- 172.21.15.78:0/1979208488 shutdown_connections 2023-10-14T11:58:51.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:51.981+0000 7fc33a385700 1 -- 172.21.15.78:0/1979208488 wait complete. 2023-10-14T11:58:51.999 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-14T11:58:52.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.171+0000 7f83a717d700 1 Processor -- start 2023-10-14T11:58:52.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.171+0000 7f83a717d700 1 -- start start 2023-10-14T11:58:52.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.171+0000 7f83a717d700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f83a006b450 0x7f83a00a9550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:52.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.171+0000 7f83a717d700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83a0072050 con 0x7f83a014fe50 2023-10-14T11:58:52.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.171+0000 7f83a717d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83a0150230 con 0x7f83a0152dd0 2023-10-14T11:58:52.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.171+0000 7f83a717d700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83a00a87c0 con 0x7f83a006b450 2023-10-14T11:58:52.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.171+0000 7f839ffff700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f83a006b450 0x7f83a00a9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:52.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.171+0000 7f83a4f19700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f83a014fe50 0x7f83a006f410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50816/0 (socket says 172.21.15.78:50816) 2023-10-14T11:58:52.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.171+0000 7f83a4f19700 1 -- 172.21.15.78:0/4231879002 learned_addr learned my addr 172.21.15.78:0/4231879002 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:52.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.171+0000 7f839f7fe700 1 -- 172.21.15.78:0/4231879002 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 230714775 0 0) 0x7f83a0072050 con 0x7f83a014fe50 2023-10-14T11:58:52.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.171+0000 7f839f7fe700 1 -- 172.21.15.78:0/4231879002 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83900035b0 con 0x7f83a014fe50 2023-10-14T11:58:52.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.171+0000 7f839ffff700 1 -- 172.21.15.78:0/4231879002 >> v1:172.21.15.195:6789/0 conn(0x7f83a0152dd0 legacy=0x7f83a015d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:52.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.171+0000 7f839ffff700 1 -- 172.21.15.78:0/4231879002 >> v1:172.21.15.78:6789/0 conn(0x7f83a014fe50 legacy=0x7f83a006f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:52.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.171+0000 7f839ffff700 1 -- 172.21.15.78:0/4231879002 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a00722a0 con 0x7f83a006b450 2023-10-14T11:58:52.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.171+0000 7f839ffff700 1 --2- 172.21.15.78:0/4231879002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83a006b450 0x7f83a00a9550 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f838c009840 tx=0x7f838c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45fbee56a5ca76e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:52.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.171+0000 7f839f7fe700 1 -- 172.21.15.78:0/4231879002 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f838c004230 con 0x7f83a006b450 2023-10-14T11:58:52.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.171+0000 7f839f7fe700 1 -- 172.21.15.78:0/4231879002 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f838c004390 con 0x7f83a006b450 2023-10-14T11:58:52.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.171+0000 7f839f7fe700 1 -- 172.21.15.78:0/4231879002 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f838c0036c0 con 0x7f83a006b450 2023-10-14T11:58:52.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.172+0000 7f83a717d700 1 -- 172.21.15.78:0/4231879002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83a006b450 msgr2=0x7f83a00a9550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:52.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.172+0000 7f83a717d700 1 --2- 172.21.15.78:0/4231879002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83a006b450 0x7f83a00a9550 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f838c009840 tx=0x7f838c009b50 comp rx=0 tx=0).stop 2023-10-14T11:58:52.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.172+0000 7f83a717d700 1 -- 172.21.15.78:0/4231879002 shutdown_connections 2023-10-14T11:58:52.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.172+0000 7f83a717d700 1 --2- 172.21.15.78:0/4231879002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83a006b450 0x7f83a00a9550 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:52.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.172+0000 7f83a717d700 1 -- 172.21.15.78:0/4231879002 >> 172.21.15.78:0/4231879002 conn(0x7f83a00712f0 msgr2=0x7f83a0071700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:52.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.172+0000 7f83a717d700 1 -- 172.21.15.78:0/4231879002 shutdown_connections 2023-10-14T11:58:52.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.172+0000 7f83a717d700 1 -- 172.21.15.78:0/4231879002 wait complete. 2023-10-14T11:58:52.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.173+0000 7f83a717d700 1 Processor -- start 2023-10-14T11:58:52.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.173+0000 7f83a717d700 1 -- start start 2023-10-14T11:58:52.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.174+0000 7f83a717d700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f83a006b450 0x7f83a00c0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:52.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.174+0000 7f83a717d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83a014fe50 0x7f83a0141590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:52.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.174+0000 7f83a717d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f83a0152dd0 0x7f83a0141ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:52.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.174+0000 7f83a717d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a01531e0 con 0x7f83a014fe50 2023-10-14T11:58:52.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.174+0000 7f83a717d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a01e9fc0 con 0x7f83a0152dd0 2023-10-14T11:58:52.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.174+0000 7f83a717d700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83a00aae10 con 0x7f83a006b450 2023-10-14T11:58:52.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.174+0000 7f839ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83a014fe50 0x7f83a0141590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:52.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.174+0000 7f839ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83a014fe50 0x7f83a0141590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45946/0 (socket says 172.21.15.78:45946) 2023-10-14T11:58:52.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.174+0000 7f839ffff700 1 -- 172.21.15.78:0/3188899661 learned_addr learned my addr 172.21.15.78:0/3188899661 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:52.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.174+0000 7f83a571a700 1 --2- 172.21.15.78:0/3188899661 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f83a0152dd0 0x7f83a0141ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:52.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.174+0000 7f83a4f19700 1 --2- 172.21.15.78:0/3188899661 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f83a006b450 0x7f83a00c0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:52.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.174+0000 7f839ffff700 1 -- 172.21.15.78:0/3188899661 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f83a006b450 msgr2=0x7f83a00c0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:52.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.174+0000 7f839ffff700 1 --2- 172.21.15.78:0/3188899661 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f83a006b450 0x7f83a00c0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:52.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.174+0000 7f839ffff700 1 -- 172.21.15.78:0/3188899661 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f83a0152dd0 msgr2=0x7f83a0141ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:52.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.174+0000 7f839ffff700 1 --2- 172.21.15.78:0/3188899661 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f83a0152dd0 0x7f83a0141ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:52.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.174+0000 7f839ffff700 1 -- 172.21.15.78:0/3188899661 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f838c0094a0 con 0x7f83a014fe50 2023-10-14T11:58:52.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.174+0000 7f839ffff700 1 --2- 172.21.15.78:0/3188899661 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83a014fe50 0x7f83a0141590 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f838c0090b0 tx=0x7f838c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:52.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.175+0000 7f839d7fa700 1 -- 172.21.15.78:0/3188899661 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f838c011630 con 0x7f83a014fe50 2023-10-14T11:58:52.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.175+0000 7f839d7fa700 1 -- 172.21.15.78:0/3188899661 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f838c011790 con 0x7f83a014fe50 2023-10-14T11:58:52.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.175+0000 7f839d7fa700 1 -- 172.21.15.78:0/3188899661 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f838c011b40 con 0x7f83a014fe50 2023-10-14T11:58:52.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.175+0000 7f83a717d700 1 -- 172.21.15.78:0/3188899661 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83a006d5c0 con 0x7f83a014fe50 2023-10-14T11:58:52.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.175+0000 7f83a717d700 1 -- 172.21.15.78:0/3188899661 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83a01536a0 con 0x7f83a014fe50 2023-10-14T11:58:52.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.176+0000 7f83a717d700 1 -- 172.21.15.78:0/3188899661 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83a01ea130 con 0x7f83a014fe50 2023-10-14T11:58:52.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.176+0000 7f839d7fa700 1 -- 172.21.15.78:0/3188899661 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f838c018040 con 0x7f83a014fe50 2023-10-14T11:58:52.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.176+0000 7f839d7fa700 1 --2- 172.21.15.78:0/3188899661 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f838406af00 0x7f838406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:52.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.176+0000 7f839d7fa700 1 -- 172.21.15.78:0/3188899661 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f838c089ed0 con 0x7f83a014fe50 2023-10-14T11:58:52.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.176+0000 7f83a4f19700 1 --2- 172.21.15.78:0/3188899661 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f838406af00 0x7f838406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:52.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.177+0000 7f83a4f19700 1 --2- 172.21.15.78:0/3188899661 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f838406af00 0x7f838406d3c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8394005910 tx=0x7f8394009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:52.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.181+0000 7f839d7fa700 1 -- 172.21.15.78:0/3188899661 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f83a01ea130 con 0x7f83a014fe50 2023-10-14T11:58:52.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:52.371+0000 7f83a717d700 1 -- 172.21.15.78:0/3188899661 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f83a01ea130 con 0x7f83a014fe50 2023-10-14T11:58:53.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.127+0000 7f839d7fa700 1 -- 172.21.15.78:0/3188899661 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v23) v1 ==== 181+0+0 (secure 0 0 0) 0x7f83a01ea130 con 0x7f83a014fe50 2023-10-14T11:58:53.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.131+0000 7f83a717d700 1 -- 172.21.15.78:0/3188899661 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f838406af00 msgr2=0x7f838406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:53.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.131+0000 7f83a717d700 1 --2- 172.21.15.78:0/3188899661 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f838406af00 0x7f838406d3c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8394005910 tx=0x7f8394009040 comp rx=0 tx=0).stop 2023-10-14T11:58:53.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.131+0000 7f83a717d700 1 -- 172.21.15.78:0/3188899661 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83a014fe50 msgr2=0x7f83a0141590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:53.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.131+0000 7f83a717d700 1 --2- 172.21.15.78:0/3188899661 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83a014fe50 0x7f83a0141590 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f838c0090b0 tx=0x7f838c004750 comp rx=0 tx=0).stop 2023-10-14T11:58:53.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.131+0000 7f83a717d700 1 -- 172.21.15.78:0/3188899661 shutdown_connections 2023-10-14T11:58:53.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.131+0000 7f83a717d700 1 --2- 172.21.15.78:0/3188899661 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f838406af00 0x7f838406d3c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:53.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.131+0000 7f83a717d700 1 --2- 172.21.15.78:0/3188899661 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f83a0152dd0 0x7f83a0141ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:53.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.131+0000 7f83a717d700 1 --2- 172.21.15.78:0/3188899661 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f83a006b450 0x7f83a00c0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:53.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.131+0000 7f83a717d700 1 --2- 172.21.15.78:0/3188899661 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83a014fe50 0x7f83a0141590 secure :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f838c0090b0 tx=0x7f838c004750 comp rx=0 tx=0).stop 2023-10-14T11:58:53.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.131+0000 7f83a717d700 1 -- 172.21.15.78:0/3188899661 >> 172.21.15.78:0/3188899661 conn(0x7f83a00712f0 msgr2=0x7f83a014bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:53.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.131+0000 7f83a717d700 1 -- 172.21.15.78:0/3188899661 shutdown_connections 2023-10-14T11:58:53.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.131+0000 7f83a717d700 1 -- 172.21.15.78:0/3188899661 wait complete. 2023-10-14T11:58:53.136 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' removed 2023-10-14T11:58:53.146 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-14T11:58:53.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.272+0000 7f456a99a700 1 Processor -- start 2023-10-14T11:58:53.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.272+0000 7f456a99a700 1 -- start start 2023-10-14T11:58:53.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.272+0000 7f456a99a700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f45640ad7c0 0x7f45640c5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:53.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.272+0000 7f456a99a700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f456406ff10 0x7f4564153cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:53.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.272+0000 7f456a99a700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45640c5070 con 0x7f456406f330 2023-10-14T11:58:53.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.272+0000 7f456a99a700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45640c4840 con 0x7f45640ad7c0 2023-10-14T11:58:53.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.272+0000 7f456a99a700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f456406f8f0 con 0x7f456406ff10 2023-10-14T11:58:53.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.273+0000 7f4568f37700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f456406f330 0x7f4564155dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50832/0 (socket says 172.21.15.78:50832) 2023-10-14T11:58:53.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.273+0000 7f4568f37700 1 -- 172.21.15.78:0/2403071392 learned_addr learned my addr 172.21.15.78:0/2403071392 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:53.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.273+0000 7f4563fff700 1 --2- 172.21.15.78:0/2403071392 >> v2:172.21.15.78:3300/0 conn(0x7f45640ad7c0 0x7f45640c5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:53.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.273+0000 7f45637fe700 1 --2- 172.21.15.78:0/2403071392 >> v2:172.21.15.195:3300/0 conn(0x7f456406ff10 0x7f4564153cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:53.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.273+0000 7f4562ffd700 1 -- 172.21.15.78:0/2403071392 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 75102730 0 0) 0x7f45640c5070 con 0x7f456406f330 2023-10-14T11:58:53.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.273+0000 7f4562ffd700 1 -- 172.21.15.78:0/2403071392 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f454c0035b0 con 0x7f456406f330 2023-10-14T11:58:53.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.273+0000 7f4563fff700 1 -- 172.21.15.78:0/2403071392 >> v2:172.21.15.195:3300/0 conn(0x7f456406ff10 msgr2=0x7f4564153cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:53.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.273+0000 7f4563fff700 1 --2- 172.21.15.78:0/2403071392 >> v2:172.21.15.195:3300/0 conn(0x7f456406ff10 0x7f4564153cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:53.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.273+0000 7f4563fff700 1 -- 172.21.15.78:0/2403071392 >> v1:172.21.15.78:6789/0 conn(0x7f456406f330 legacy=0x7f4564155dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:53.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.273+0000 7f4563fff700 1 -- 172.21.15.78:0/2403071392 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45640c4a20 con 0x7f45640ad7c0 2023-10-14T11:58:53.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.273+0000 7f45637fe700 1 --2- 172.21.15.78:0/2403071392 >> v2:172.21.15.195:3300/0 conn(0x7f456406ff10 0x7f4564153cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:53.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.273+0000 7f4563fff700 1 --2- 172.21.15.78:0/2403071392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45640ad7c0 0x7f45640c5500 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4564072330 tx=0x7f4554009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37a3b8415c9d83ce server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:53.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.273+0000 7f4562ffd700 1 -- 172.21.15.78:0/2403071392 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4554011830 con 0x7f45640ad7c0 2023-10-14T11:58:53.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.273+0000 7f4562ffd700 1 -- 172.21.15.78:0/2403071392 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4554011990 con 0x7f45640ad7c0 2023-10-14T11:58:53.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.273+0000 7f4562ffd700 1 -- 172.21.15.78:0/2403071392 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4554011d20 con 0x7f45640ad7c0 2023-10-14T11:58:53.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.274+0000 7f456a99a700 1 -- 172.21.15.78:0/2403071392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45640ad7c0 msgr2=0x7f45640c5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:53.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.274+0000 7f456a99a700 1 --2- 172.21.15.78:0/2403071392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45640ad7c0 0x7f45640c5500 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4564072330 tx=0x7f4554009b20 comp rx=0 tx=0).stop 2023-10-14T11:58:53.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.274+0000 7f456a99a700 1 -- 172.21.15.78:0/2403071392 shutdown_connections 2023-10-14T11:58:53.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.274+0000 7f456a99a700 1 --2- 172.21.15.78:0/2403071392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45640ad7c0 0x7f45640c5500 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:53.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.274+0000 7f456a99a700 1 --2- 172.21.15.78:0/2403071392 >> v2:172.21.15.195:3300/0 conn(0x7f456406ff10 0x7f4564153cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:53.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.274+0000 7f456a99a700 1 -- 172.21.15.78:0/2403071392 >> 172.21.15.78:0/2403071392 conn(0x7f4564072770 msgr2=0x7f45640733e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:53.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.274+0000 7f456a99a700 1 -- 172.21.15.78:0/2403071392 shutdown_connections 2023-10-14T11:58:53.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.274+0000 7f456a99a700 1 -- 172.21.15.78:0/2403071392 wait complete. 2023-10-14T11:58:53.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.275+0000 7f456a99a700 1 Processor -- start 2023-10-14T11:58:53.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.275+0000 7f456a99a700 1 -- start start 2023-10-14T11:58:53.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.276+0000 7f456a99a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f456406f330 0x7f45641ed770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:53.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.276+0000 7f456a99a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f456406ff10 0x7f45640be4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:53.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.276+0000 7f456a99a700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f45640ad7c0 0x7f456406cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:53.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.276+0000 7f456a99a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45641565d0 con 0x7f456406f330 2023-10-14T11:58:53.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.276+0000 7f456a99a700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45640be020 con 0x7f456406ff10 2023-10-14T11:58:53.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.276+0000 7f456a99a700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45641562f0 con 0x7f45640ad7c0 2023-10-14T11:58:53.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.276+0000 7f4563fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f456406f330 0x7f45641ed770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:53.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.276+0000 7f4563fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f456406f330 0x7f45641ed770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45960/0 (socket says 172.21.15.78:45960) 2023-10-14T11:58:53.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.276+0000 7f4563fff700 1 -- 172.21.15.78:0/2454597502 learned_addr learned my addr 172.21.15.78:0/2454597502 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:53.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.276+0000 7f4568f37700 1 --2- 172.21.15.78:0/2454597502 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f45640ad7c0 0x7f456406cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:53.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.276+0000 7f45637fe700 1 --2- 172.21.15.78:0/2454597502 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f456406ff10 0x7f45640be4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:53.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.276+0000 7f4563fff700 1 -- 172.21.15.78:0/2454597502 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f45640ad7c0 msgr2=0x7f456406cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:53.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.276+0000 7f4563fff700 1 --2- 172.21.15.78:0/2454597502 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f45640ad7c0 0x7f456406cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:53.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.276+0000 7f4563fff700 1 -- 172.21.15.78:0/2454597502 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f456406ff10 msgr2=0x7f45640be4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:53.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.276+0000 7f4563fff700 1 --2- 172.21.15.78:0/2454597502 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f456406ff10 0x7f45640be4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:53.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.276+0000 7f4563fff700 1 -- 172.21.15.78:0/2454597502 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45540094a0 con 0x7f456406f330 2023-10-14T11:58:53.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.276+0000 7f4563fff700 1 --2- 172.21.15.78:0/2454597502 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f456406f330 0x7f45641ed770 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4554000c00 tx=0x7f4554004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:53.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.276+0000 7f4560ff9700 1 -- 172.21.15.78:0/2454597502 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45540036a0 con 0x7f456406f330 2023-10-14T11:58:53.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.276+0000 7f4560ff9700 1 -- 172.21.15.78:0/2454597502 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f455401e070 con 0x7f456406f330 2023-10-14T11:58:53.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.276+0000 7f456a99a700 1 -- 172.21.15.78:0/2454597502 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45641ee0d0 con 0x7f456406f330 2023-10-14T11:58:53.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.276+0000 7f4560ff9700 1 -- 172.21.15.78:0/2454597502 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4554003a60 con 0x7f456406f330 2023-10-14T11:58:53.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.276+0000 7f456a99a700 1 -- 172.21.15.78:0/2454597502 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f456406fa70 con 0x7f456406f330 2023-10-14T11:58:53.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.277+0000 7f456a99a700 1 -- 172.21.15.78:0/2454597502 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45640be300 con 0x7f456406f330 2023-10-14T11:58:53.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.278+0000 7f4560ff9700 1 -- 172.21.15.78:0/2454597502 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4554003bc0 con 0x7f456406f330 2023-10-14T11:58:53.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.278+0000 7f4560ff9700 1 --2- 172.21.15.78:0/2454597502 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f455006af00 0x7f455006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:53.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.278+0000 7f4560ff9700 1 -- 172.21.15.78:0/2454597502 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f455408a3e0 con 0x7f456406f330 2023-10-14T11:58:53.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.278+0000 7f45637fe700 1 --2- 172.21.15.78:0/2454597502 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f455006af00 0x7f455006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:53.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.279+0000 7f45637fe700 1 --2- 172.21.15.78:0/2454597502 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f455006af00 0x7f455006d3c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f45640c76b0 tx=0x7f4558008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:53.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.282+0000 7f4560ff9700 1 -- 172.21.15.78:0/2454597502 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4554055680 con 0x7f456406f330 2023-10-14T11:58:53.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:53.474+0000 7f456a99a700 1 -- 172.21.15.78:0/2454597502 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4564074b80 con 0x7f456406f330 2023-10-14T11:58:53.551 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T11:58:53.553 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T11:58:54.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.132+0000 7f4560ff9700 1 -- 172.21.15.78:0/2454597502 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v24) v1 ==== 169+0+0 (secure 0 0 0) 0x7f4554058ca0 con 0x7f456406f330 2023-10-14T11:58:54.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.135+0000 7f456a99a700 1 -- 172.21.15.78:0/2454597502 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f455006af00 msgr2=0x7f455006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:54.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.135+0000 7f456a99a700 1 --2- 172.21.15.78:0/2454597502 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f455006af00 0x7f455006d3c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f45640c76b0 tx=0x7f4558008040 comp rx=0 tx=0).stop 2023-10-14T11:58:54.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.135+0000 7f456a99a700 1 -- 172.21.15.78:0/2454597502 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f456406f330 msgr2=0x7f45641ed770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:54.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.135+0000 7f456a99a700 1 --2- 172.21.15.78:0/2454597502 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f456406f330 0x7f45641ed770 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4554000c00 tx=0x7f4554004750 comp rx=0 tx=0).stop 2023-10-14T11:58:54.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.135+0000 7f456a99a700 1 -- 172.21.15.78:0/2454597502 shutdown_connections 2023-10-14T11:58:54.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.135+0000 7f456a99a700 1 --2- 172.21.15.78:0/2454597502 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f45640ad7c0 0x7f456406cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:54.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.135+0000 7f456a99a700 1 --2- 172.21.15.78:0/2454597502 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f456406f330 0x7f45641ed770 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:54.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.135+0000 7f456a99a700 1 --2- 172.21.15.78:0/2454597502 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f455006af00 0x7f455006d3c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:54.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.135+0000 7f456a99a700 1 --2- 172.21.15.78:0/2454597502 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f456406ff10 0x7f45640be4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:54.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.135+0000 7f456a99a700 1 -- 172.21.15.78:0/2454597502 >> 172.21.15.78:0/2454597502 conn(0x7f4564072770 msgr2=0x7f45640c6180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:54.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.135+0000 7f456a99a700 1 -- 172.21.15.78:0/2454597502 shutdown_connections 2023-10-14T11:58:54.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.135+0000 7f456a99a700 1 -- 172.21.15.78:0/2454597502 wait complete. 2023-10-14T11:58:54.139 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' removed 2023-10-14T11:58:54.150 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-14T11:58:54.151 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-14T11:58:54.151 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-14T11:58:54.151 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T11:58:54.152 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-14T11:58:54.152 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-14T11:58:54.152 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-14T11:58:54.153 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T11:58:54.153 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-14T11:58:54.153 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-14T11:58:54.154 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-14T11:58:54.154 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T11:58:54.154 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-14T11:58:54.154 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-14T11:58:54.155 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-14T11:58:54.155 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T11:58:54.161 INFO:tasks.ceph.mds.a:Stopped 2023-10-14T11:58:54.162 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-14T11:58:54.164 INFO:tasks.ceph.mds.c:Stopped 2023-10-14T11:58:54.164 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-14T11:58:54.165 INFO:tasks.ceph.mds.b:Stopped 2023-10-14T11:58:54.165 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-14T11:58:54.166 INFO:tasks.ceph.mds.d:Stopped 2023-10-14T11:58:54.167 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-14T11:58:54.205 INFO:tasks.ceph.mds.a:Started 2023-10-14T11:58:54.206 DEBUG:teuthology.parallel:result is None 2023-10-14T11:58:54.207 INFO:tasks.ceph.mds.c:Started 2023-10-14T11:58:54.208 DEBUG:teuthology.parallel:result is None 2023-10-14T11:58:54.208 INFO:tasks.ceph.mds.b:Started 2023-10-14T11:58:54.209 DEBUG:teuthology.parallel:result is None 2023-10-14T11:58:54.209 INFO:tasks.ceph.mds.d:Started 2023-10-14T11:58:54.210 DEBUG:teuthology.parallel:result is None 2023-10-14T11:58:54.210 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-14T11:58:54.285 INFO:tasks.ceph.mds.a.smithi078.stdout:starting mds.a at 2023-10-14T11:58:54.285 INFO:tasks.ceph.mds.c.smithi078.stdout:starting mds.c at 2023-10-14T11:58:54.293 INFO:tasks.ceph.mds.b.smithi195.stdout:starting mds.b at 2023-10-14T11:58:54.293 INFO:tasks.ceph.mds.d.smithi195.stdout:starting mds.d at 2023-10-14T11:58:54.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.343+0000 7f7a5c0e9700 1 Processor -- start 2023-10-14T11:58:54.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.344+0000 7f7a5c0e9700 1 -- start start 2023-10-14T11:58:54.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.345+0000 7f7a5c0e9700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f7a54072b40 0x7f7a54152440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:54.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.345+0000 7f7a5c0e9700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a540712f0 con 0x7f7a54072670 2023-10-14T11:58:54.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.345+0000 7f7a5c0e9700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a540715d0 con 0x7f7a540a5180 2023-10-14T11:58:54.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.345+0000 7f7a5c0e9700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a540707c0 con 0x7f7a54072b40 2023-10-14T11:58:54.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.345+0000 7f7a59684700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f7a54072670 0x7f7a540c3db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50866/0 (socket says 172.21.15.78:50866) 2023-10-14T11:58:54.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.345+0000 7f7a59684700 1 -- 172.21.15.78:0/1026936576 learned_addr learned my addr 172.21.15.78:0/1026936576 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:54.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.345+0000 7f7a5a686700 1 --2- 172.21.15.78:0/1026936576 >> v2:172.21.15.195:3300/0 conn(0x7f7a54072b40 0x7f7a54152440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:54.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.345+0000 7f7a5a686700 1 -- 172.21.15.78:0/1026936576 >> v1:172.21.15.195:6789/0 conn(0x7f7a540a5180 legacy=0x7f7a5406d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:54.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.345+0000 7f7a5a686700 1 -- 172.21.15.78:0/1026936576 >> v1:172.21.15.78:6789/0 conn(0x7f7a54072670 legacy=0x7f7a540c3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:54.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.345+0000 7f7a5a686700 1 -- 172.21.15.78:0/1026936576 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a540718b0 con 0x7f7a54072b40 2023-10-14T11:58:54.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.346+0000 7f7a5a686700 1 --2- 172.21.15.78:0/1026936576 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7a54072b40 0x7f7a54152440 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c009840 tx=0x7f7a4c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c46de27702acc93 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:54.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.346+0000 7f7a58e83700 1 -- 172.21.15.78:0/1026936576 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c004230 con 0x7f7a54072b40 2023-10-14T11:58:54.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.346+0000 7f7a58e83700 1 -- 172.21.15.78:0/1026936576 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a4c004390 con 0x7f7a54072b40 2023-10-14T11:58:54.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.346+0000 7f7a58e83700 1 -- 172.21.15.78:0/1026936576 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c0036a0 con 0x7f7a54072b40 2023-10-14T11:58:54.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.347+0000 7f7a5c0e9700 1 -- 172.21.15.78:0/1026936576 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7a54072b40 msgr2=0x7f7a54152440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:54.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.347+0000 7f7a5c0e9700 1 --2- 172.21.15.78:0/1026936576 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7a54072b40 0x7f7a54152440 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c009840 tx=0x7f7a4c009b50 comp rx=0 tx=0).stop 2023-10-14T11:58:54.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.347+0000 7f7a5c0e9700 1 -- 172.21.15.78:0/1026936576 shutdown_connections 2023-10-14T11:58:54.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.347+0000 7f7a5c0e9700 1 --2- 172.21.15.78:0/1026936576 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7a54072b40 0x7f7a54152440 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:54.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.347+0000 7f7a5c0e9700 1 -- 172.21.15.78:0/1026936576 >> 172.21.15.78:0/1026936576 conn(0x7f7a5406af90 msgr2=0x7f7a540c5eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:54.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.347+0000 7f7a5c0e9700 1 -- 172.21.15.78:0/1026936576 shutdown_connections 2023-10-14T11:58:54.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.347+0000 7f7a5c0e9700 1 -- 172.21.15.78:0/1026936576 wait complete. 2023-10-14T11:58:54.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.348+0000 7f7a5c0e9700 1 Processor -- start 2023-10-14T11:58:54.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.348+0000 7f7a5c0e9700 1 -- start start 2023-10-14T11:58:54.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.349+0000 7f7a5c0e9700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7a54072670 0x7f7a540b8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:54.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.349+0000 7f7a5c0e9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a54072b40 0x7f7a540b8550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:54.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.349+0000 7f7a5c0e9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7a540a5180 0x7f7a540bcef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:54.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.349+0000 7f7a5c0e9700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a54155d80 con 0x7f7a54072b40 2023-10-14T11:58:54.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.349+0000 7f7a5c0e9700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a540bc7d0 con 0x7f7a540a5180 2023-10-14T11:58:54.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.349+0000 7f7a5c0e9700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a540bfbf0 con 0x7f7a54072670 2023-10-14T11:58:54.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.349+0000 7f7a59684700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a54072b40 0x7f7a540b8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:54.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.349+0000 7f7a59684700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a54072b40 0x7f7a540b8550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45998/0 (socket says 172.21.15.78:45998) 2023-10-14T11:58:54.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.349+0000 7f7a59684700 1 -- 172.21.15.78:0/1249612859 learned_addr learned my addr 172.21.15.78:0/1249612859 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:54.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.349+0000 7f7a5a686700 1 --2- 172.21.15.78:0/1249612859 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7a540a5180 0x7f7a540bcef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:54.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.349+0000 7f7a59e85700 1 --2- 172.21.15.78:0/1249612859 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7a54072670 0x7f7a540b8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:54.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.349+0000 7f7a59684700 1 -- 172.21.15.78:0/1249612859 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7a54072670 msgr2=0x7f7a540b8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:54.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.349+0000 7f7a59684700 1 --2- 172.21.15.78:0/1249612859 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7a54072670 0x7f7a540b8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:54.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.349+0000 7f7a59684700 1 -- 172.21.15.78:0/1249612859 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7a540a5180 msgr2=0x7f7a540bcef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:54.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.349+0000 7f7a59684700 1 --2- 172.21.15.78:0/1249612859 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7a540a5180 0x7f7a540bcef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:54.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.349+0000 7f7a59684700 1 -- 172.21.15.78:0/1249612859 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a4c0094a0 con 0x7f7a54072b40 2023-10-14T11:58:54.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.350+0000 7f7a59684700 1 --2- 172.21.15.78:0/1249612859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a54072b40 0x7f7a540b8550 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7a4400c4c0 tx=0x7f7a4400c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:54.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.350+0000 7f7a4a7fc700 1 -- 172.21.15.78:0/1249612859 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a44005d70 con 0x7f7a54072b40 2023-10-14T11:58:54.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.350+0000 7f7a4a7fc700 1 -- 172.21.15.78:0/1249612859 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a4400ce90 con 0x7f7a54072b40 2023-10-14T11:58:54.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.350+0000 7f7a4a7fc700 1 -- 172.21.15.78:0/1249612859 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a440072f0 con 0x7f7a54072b40 2023-10-14T11:58:54.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.350+0000 7f7a5c0e9700 1 -- 172.21.15.78:0/1249612859 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a541557c0 con 0x7f7a54072b40 2023-10-14T11:58:54.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.350+0000 7f7a5c0e9700 1 -- 172.21.15.78:0/1249612859 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a54154c30 con 0x7f7a54072b40 2023-10-14T11:58:54.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.351+0000 7f7a5c0e9700 1 -- 172.21.15.78:0/1249612859 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a54154a90 con 0x7f7a54072b40 2023-10-14T11:58:54.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.351+0000 7f7a4a7fc700 1 -- 172.21.15.78:0/1249612859 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7a44007450 con 0x7f7a54072b40 2023-10-14T11:58:54.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.352+0000 7f7a4a7fc700 1 --2- 172.21.15.78:0/1249612859 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7a3806af00 0x7f7a3806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:54.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.352+0000 7f7a4a7fc700 1 -- 172.21.15.78:0/1249612859 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7a44088640 con 0x7f7a54072b40 2023-10-14T11:58:54.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.352+0000 7f7a59e85700 1 --2- 172.21.15.78:0/1249612859 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7a3806af00 0x7f7a3806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:54.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.352+0000 7f7a59e85700 1 --2- 172.21.15.78:0/1249612859 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7a3806af00 0x7f7a3806d3c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7a541622c0 tx=0x7f7a5000d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:54.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.356+0000 7f7a4a7fc700 1 -- 172.21.15.78:0/1249612859 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7a54154a90 con 0x7f7a54072b40 2023-10-14T11:58:54.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:54.560+0000 7f7a5c0e9700 1 -- 172.21.15.78:0/1249612859 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f7a54154a90 con 0x7f7a54072b40 2023-10-14T11:58:55.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.144+0000 7f7a4a7fc700 1 -- 172.21.15.78:0/1249612859 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v26) v1 ==== 97+0+0 (secure 0 0 0) 0x7f7a54154a90 con 0x7f7a54072b40 2023-10-14T11:58:55.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.148+0000 7f7a5c0e9700 1 -- 172.21.15.78:0/1249612859 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7a3806af00 msgr2=0x7f7a3806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:55.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.148+0000 7f7a5c0e9700 1 --2- 172.21.15.78:0/1249612859 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7a3806af00 0x7f7a3806d3c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7a541622c0 tx=0x7f7a5000d040 comp rx=0 tx=0).stop 2023-10-14T11:58:55.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.148+0000 7f7a5c0e9700 1 -- 172.21.15.78:0/1249612859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a54072b40 msgr2=0x7f7a540b8550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:55.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.148+0000 7f7a5c0e9700 1 --2- 172.21.15.78:0/1249612859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a54072b40 0x7f7a540b8550 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7a4400c4c0 tx=0x7f7a4400c7d0 comp rx=0 tx=0).stop 2023-10-14T11:58:55.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.148+0000 7f7a5c0e9700 1 -- 172.21.15.78:0/1249612859 shutdown_connections 2023-10-14T11:58:55.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.148+0000 7f7a5c0e9700 1 --2- 172.21.15.78:0/1249612859 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7a3806af00 0x7f7a3806d3c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:55.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.148+0000 7f7a5c0e9700 1 --2- 172.21.15.78:0/1249612859 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7a540a5180 0x7f7a540bcef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:55.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.148+0000 7f7a5c0e9700 1 --2- 172.21.15.78:0/1249612859 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7a54072670 0x7f7a540b8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:55.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.148+0000 7f7a5c0e9700 1 --2- 172.21.15.78:0/1249612859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a54072b40 0x7f7a540b8550 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:55.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.148+0000 7f7a5c0e9700 1 -- 172.21.15.78:0/1249612859 >> 172.21.15.78:0/1249612859 conn(0x7f7a5406af90 msgr2=0x7f7a5414f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:55.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.148+0000 7f7a5c0e9700 1 -- 172.21.15.78:0/1249612859 shutdown_connections 2023-10-14T11:58:55.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.148+0000 7f7a5c0e9700 1 -- 172.21.15.78:0/1249612859 wait complete. 2023-10-14T11:58:55.151 INFO:teuthology.orchestra.run.smithi078.stderr: removed all blocklist entries 2023-10-14T11:58:55.165 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-14T11:58:55.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.290+0000 7f63bcfb4700 1 Processor -- start 2023-10-14T11:58:55.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.290+0000 7f63bcfb4700 1 -- start start 2023-10-14T11:58:55.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.291+0000 7f63bcfb4700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f63b80c72a0 0x7f63b80c7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:55.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.291+0000 7f63bcfb4700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63b8156b20 con 0x7f63b80c44c0 2023-10-14T11:58:55.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.291+0000 7f63bcfb4700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63b8155f10 con 0x7f63b80c2c90 2023-10-14T11:58:55.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.291+0000 7f63bcfb4700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63b81561f0 con 0x7f63b80c72a0 2023-10-14T11:58:55.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.291+0000 7f63b659c700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f63b80c44c0 0x7f63b80c86e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50880/0 (socket says 172.21.15.78:50880) 2023-10-14T11:58:55.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.291+0000 7f63b659c700 1 -- 172.21.15.78:0/1101264736 learned_addr learned my addr 172.21.15.78:0/1101264736 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:55.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.291+0000 7f63b559a700 1 -- 172.21.15.78:0/1101264736 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 248000669 0 0) 0x7f63b8156b20 con 0x7f63b80c44c0 2023-10-14T11:58:55.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.291+0000 7f63b559a700 1 -- 172.21.15.78:0/1101264736 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63a00035b0 con 0x7f63b80c44c0 2023-10-14T11:58:55.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.291+0000 7f63b5d9b700 1 --2- 172.21.15.78:0/1101264736 >> v2:172.21.15.195:3300/0 conn(0x7f63b80c72a0 0x7f63b80c7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:55.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.291+0000 7f63b559a700 1 -- 172.21.15.78:0/1101264736 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3192279423 0 0) 0x7f63a00035b0 con 0x7f63b80c44c0 2023-10-14T11:58:55.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.291+0000 7f63b559a700 1 -- 172.21.15.78:0/1101264736 >> v2:172.21.15.195:3300/0 conn(0x7f63b80c72a0 msgr2=0x7f63b80c7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:55.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.291+0000 7f63b559a700 1 --2- 172.21.15.78:0/1101264736 >> v2:172.21.15.195:3300/0 conn(0x7f63b80c72a0 0x7f63b80c7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:55.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.291+0000 7f63b559a700 1 -- 172.21.15.78:0/1101264736 >> v1:172.21.15.195:6789/0 conn(0x7f63b80c2c90 legacy=0x7f63b80c30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:55.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.291+0000 7f63b559a700 1 -- 172.21.15.78:0/1101264736 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b81564d0 con 0x7f63b80c44c0 2023-10-14T11:58:55.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.291+0000 7f63b559a700 1 -- 172.21.15.78:0/1101264736 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441799917 0 0) 0x7f63ac001670 con 0x7f63b80c44c0 2023-10-14T11:58:55.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.291+0000 7f63b559a700 1 -- 172.21.15.78:0/1101264736 >> v1:172.21.15.78:6789/0 conn(0x7f63b80c44c0 legacy=0x7f63b80c86e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:55.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.291+0000 7f63b559a700 1 --2- 172.21.15.78:0/1101264736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63a0003a00 0x7f63a0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:55.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.291+0000 7f63b559a700 1 --2- 172.21.15.78:0/1101264736 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f63a0004850 0x7f63a0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:55.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.291+0000 7f63b559a700 1 --2- 172.21.15.78:0/1101264736 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63a0005210 0x7f63a000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:55.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.291+0000 7f63b559a700 1 -- 172.21.15.78:0/1101264736 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b81561f0 con 0x7f63a0003a00 2023-10-14T11:58:55.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.291+0000 7f63b559a700 1 -- 172.21.15.78:0/1101264736 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a000c140 con 0x7f63a0005210 2023-10-14T11:58:55.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.291+0000 7f63b559a700 1 -- 172.21.15.78:0/1101264736 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63a000c300 con 0x7f63a0004850 2023-10-14T11:58:55.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.291+0000 7f63b659c700 1 --2- 172.21.15.78:0/1101264736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63a0003a00 0x7f63a0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:55.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.292+0000 7f63b5d9b700 1 --2- 172.21.15.78:0/1101264736 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f63a0004850 0x7f63a0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:55.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.292+0000 7f63b6d9d700 1 --2- 172.21.15.78:0/1101264736 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63a0005210 0x7f63a000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:55.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.292+0000 7f63b659c700 1 -- 172.21.15.78:0/1101264736 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f63a0004850 msgr2=0x7f63a0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:55.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.292+0000 7f63b659c700 1 --2- 172.21.15.78:0/1101264736 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f63a0004850 0x7f63a0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:55.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.292+0000 7f63b659c700 1 -- 172.21.15.78:0/1101264736 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63a0005210 msgr2=0x7f63a000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:55.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.292+0000 7f63b659c700 1 --2- 172.21.15.78:0/1101264736 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63a0005210 0x7f63a000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:55.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.292+0000 7f63b6d9d700 1 -- 172.21.15.78:0/1101264736 reap_dead start 2023-10-14T11:58:55.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.292+0000 7f63b659c700 1 -- 172.21.15.78:0/1101264736 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a00035b0 con 0x7f63a0003a00 2023-10-14T11:58:55.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.292+0000 7f63b5d9b700 1 --2- 172.21.15.78:0/1101264736 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f63a0004850 0x7f63a0004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:55.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.292+0000 7f63b659c700 1 --2- 172.21.15.78:0/1101264736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63a0003a00 0x7f63a0003e00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f63ac00c210 tx=0x7f63a0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a17a50ee44e28d1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:55.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.292+0000 7f63b559a700 1 -- 172.21.15.78:0/1101264736 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ac00cdb0 con 0x7f63a0003a00 2023-10-14T11:58:55.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.292+0000 7f63b559a700 1 -- 172.21.15.78:0/1101264736 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63ac007ba0 con 0x7f63a0003a00 2023-10-14T11:58:55.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.292+0000 7f63b559a700 1 -- 172.21.15.78:0/1101264736 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ac005ba0 con 0x7f63a0003a00 2023-10-14T11:58:55.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.292+0000 7f63bcfb4700 1 -- 172.21.15.78:0/1101264736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63a0003a00 msgr2=0x7f63a0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:55.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.292+0000 7f63bcfb4700 1 --2- 172.21.15.78:0/1101264736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63a0003a00 0x7f63a0003e00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f63ac00c210 tx=0x7f63a0005880 comp rx=0 tx=0).stop 2023-10-14T11:58:55.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.293+0000 7f63bcfb4700 1 -- 172.21.15.78:0/1101264736 shutdown_connections 2023-10-14T11:58:55.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.293+0000 7f63bcfb4700 1 --2- 172.21.15.78:0/1101264736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63a0003a00 0x7f63a0003e00 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:55.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.293+0000 7f63bcfb4700 1 -- 172.21.15.78:0/1101264736 >> 172.21.15.78:0/1101264736 conn(0x7f63b80737d0 msgr2=0x7f63b8073be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:55.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.293+0000 7f63bcfb4700 1 -- 172.21.15.78:0/1101264736 shutdown_connections 2023-10-14T11:58:55.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.293+0000 7f63bcfb4700 1 -- 172.21.15.78:0/1101264736 wait complete. 2023-10-14T11:58:55.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.294+0000 7f63bcfb4700 1 Processor -- start 2023-10-14T11:58:55.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.294+0000 7f63bcfb4700 1 -- start start 2023-10-14T11:58:55.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.295+0000 7f63bcfb4700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f63a0003a00 0x7f63b80a65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:55.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.295+0000 7f63bcfb4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63b806ba90 0x7f63b806d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:55.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.295+0000 7f63bcfb4700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63b8154bd0 0x7f63b8155080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:55.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.295+0000 7f63bcfb4700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b80bc0c0 con 0x7f63b806ba90 2023-10-14T11:58:55.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.295+0000 7f63bcfb4700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b80bbdb0 con 0x7f63b8154bd0 2023-10-14T11:58:55.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.295+0000 7f63bcfb4700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63b80c2190 con 0x7f63a0003a00 2023-10-14T11:58:55.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.295+0000 7f63b5d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63b806ba90 0x7f63b806d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:55.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.295+0000 7f63b5d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63b806ba90 0x7f63b806d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46018/0 (socket says 172.21.15.78:46018) 2023-10-14T11:58:55.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.295+0000 7f63b5d9b700 1 -- 172.21.15.78:0/565339513 learned_addr learned my addr 172.21.15.78:0/565339513 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:55.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.295+0000 7f63b659c700 1 --2- 172.21.15.78:0/565339513 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f63a0003a00 0x7f63b80a65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:55.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.295+0000 7f63b6d9d700 1 --2- 172.21.15.78:0/565339513 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63b8154bd0 0x7f63b8155080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:55.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.295+0000 7f63b5d9b700 1 -- 172.21.15.78:0/565339513 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f63a0003a00 msgr2=0x7f63b80a65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:55.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.295+0000 7f63b5d9b700 1 --2- 172.21.15.78:0/565339513 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f63a0003a00 0x7f63b80a65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:55.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.295+0000 7f63b5d9b700 1 -- 172.21.15.78:0/565339513 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63b8154bd0 msgr2=0x7f63b8155080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:55.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.295+0000 7f63b5d9b700 1 --2- 172.21.15.78:0/565339513 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63b8154bd0 0x7f63b8155080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:55.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.295+0000 7f63b5d9b700 1 -- 172.21.15.78:0/565339513 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b81ea690 con 0x7f63b806ba90 2023-10-14T11:58:55.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.295+0000 7f63b6d9d700 1 --2- 172.21.15.78:0/565339513 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63b8154bd0 0x7f63b8155080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:55.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.295+0000 7f63b659c700 1 --2- 172.21.15.78:0/565339513 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f63a0003a00 0x7f63b80a65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:55.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.295+0000 7f63b5d9b700 1 --2- 172.21.15.78:0/565339513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63b806ba90 0x7f63b806d340 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f63a8010840 tx=0x7f63a8010c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:55.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.295+0000 7f63a6ffd700 1 -- 172.21.15.78:0/565339513 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a8015880 con 0x7f63b806ba90 2023-10-14T11:58:55.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.295+0000 7f63a6ffd700 1 -- 172.21.15.78:0/565339513 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63a800c050 con 0x7f63b806ba90 2023-10-14T11:58:55.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.295+0000 7f63bcfb4700 1 -- 172.21.15.78:0/565339513 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63b81ea970 con 0x7f63b806ba90 2023-10-14T11:58:55.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.295+0000 7f63a6ffd700 1 -- 172.21.15.78:0/565339513 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a8015c20 con 0x7f63b806ba90 2023-10-14T11:58:55.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.295+0000 7f63bcfb4700 1 -- 172.21.15.78:0/565339513 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63b80bc550 con 0x7f63b806ba90 2023-10-14T11:58:55.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.296+0000 7f63bcfb4700 1 -- 172.21.15.78:0/565339513 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63b8074b80 con 0x7f63b806ba90 2023-10-14T11:58:55.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.297+0000 7f63a6ffd700 1 -- 172.21.15.78:0/565339513 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f63a8016420 con 0x7f63b806ba90 2023-10-14T11:58:55.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.297+0000 7f63a6ffd700 1 --2- 172.21.15.78:0/565339513 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f639806af00 0x7f639806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:55.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.297+0000 7f63a6ffd700 1 -- 172.21.15.78:0/565339513 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f63a80896a0 con 0x7f63b806ba90 2023-10-14T11:58:55.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.297+0000 7f63b659c700 1 --2- 172.21.15.78:0/565339513 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f639806af00 0x7f639806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:55.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.297+0000 7f63b659c700 1 --2- 172.21.15.78:0/565339513 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f639806af00 0x7f639806d3c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f63ac007d50 tx=0x7f63ac007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:55.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.301+0000 7f63a6ffd700 1 -- 172.21.15.78:0/565339513 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f63a8054b30 con 0x7f63b806ba90 2023-10-14T11:58:55.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.568+0000 7f63bcfb4700 1 -- 172.21.15.78:0/565339513 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f63b80c7080 con 0x7f63b806ba90 2023-10-14T11:58:55.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.568+0000 7f63a6ffd700 1 -- 172.21.15.78:0/565339513 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v8) v1 ==== 80+0+6065 (secure 0 0 0) 0x7f63a8058150 con 0x7f63b806ba90 2023-10-14T11:58:55.569 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:55.570 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-14T11:58:55.570 INFO:teuthology.orchestra.run.smithi078.stdout: "auth_dump": [ 2023-10-14T11:58:55.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-14T11:58:55.570 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.a", 2023-10-14T11:58:55.571 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQAlgipldx40NhAA5+wjFFtoBDWvyFlYkTz5Rg==", 2023-10-14T11:58:55.571 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2023-10-14T11:58:55.571 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2023-10-14T11:58:55.571 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2023-10-14T11:58:55.571 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2023-10-14T11:58:55.572 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2023-10-14T11:58:55.572 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-14T11:58:55.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-14T11:58:55.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-14T11:58:55.572 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.b", 2023-10-14T11:58:55.573 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQAmgiplG9sSBxAAv9SDzXKMbqC9XA8nstj/lQ==", 2023-10-14T11:58:55.573 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2023-10-14T11:58:55.573 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2023-10-14T11:58:55.573 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2023-10-14T11:58:55.573 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2023-10-14T11:58:55.574 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2023-10-14T11:58:55.574 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-14T11:58:55.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-14T11:58:55.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-14T11:58:55.574 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.c", 2023-10-14T11:58:55.575 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQAmgipli8TXARAAcFS0dWo49wsfm5kaTsXnWg==", 2023-10-14T11:58:55.575 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2023-10-14T11:58:55.575 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2023-10-14T11:58:55.575 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2023-10-14T11:58:55.575 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2023-10-14T11:58:55.576 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2023-10-14T11:58:55.576 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-14T11:58:55.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-14T11:58:55.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-14T11:58:55.576 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.d", 2023-10-14T11:58:55.576 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQAmgiplSnGADhAAejccrmfYZiTxuKlX6yuIvQ==", 2023-10-14T11:58:55.577 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2023-10-14T11:58:55.577 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2023-10-14T11:58:55.577 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2023-10-14T11:58:55.577 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2023-10-14T11:58:55.577 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2023-10-14T11:58:55.578 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-14T11:58:55.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-14T11:58:55.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-14T11:58:55.578 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.0", 2023-10-14T11:58:55.578 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQApgiplJwmQGxAAXm76A5lPkboC5Ldi1/sEAw==", 2023-10-14T11:58:55.579 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2023-10-14T11:58:55.579 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2023-10-14T11:58:55.579 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2023-10-14T11:58:55.579 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2023-10-14T11:58:55.579 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-14T11:58:55.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-14T11:58:55.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-14T11:58:55.580 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.1", 2023-10-14T11:58:55.580 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQAsgiplhKZuGxAAIGuMc3wnXW5P3hZrGXDYYA==", 2023-10-14T11:58:55.580 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2023-10-14T11:58:55.580 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2023-10-14T11:58:55.581 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2023-10-14T11:58:55.581 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2023-10-14T11:58:55.581 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-14T11:58:55.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-14T11:58:55.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-14T11:58:55.582 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.2", 2023-10-14T11:58:55.582 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQAvgiplKe43GxAA0V3t/PIav9W0B6QRbGR/WQ==", 2023-10-14T11:58:55.582 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2023-10-14T11:58:55.582 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2023-10-14T11:58:55.582 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2023-10-14T11:58:55.583 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2023-10-14T11:58:55.583 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-14T11:58:55.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-14T11:58:55.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-14T11:58:55.583 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.3", 2023-10-14T11:58:55.583 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQAygiplGxOAGxAAThRNVROQeMI4y+sipJ7u1A==", 2023-10-14T11:58:55.584 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2023-10-14T11:58:55.584 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2023-10-14T11:58:55.584 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2023-10-14T11:58:55.584 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2023-10-14T11:58:55.584 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-14T11:58:55.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-14T11:58:55.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-14T11:58:55.585 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.4", 2023-10-14T11:58:55.585 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQA2giplqoTWJxAA48WUfi3wb40Z6q8YGsQIBQ==", 2023-10-14T11:58:55.585 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2023-10-14T11:58:55.586 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2023-10-14T11:58:55.586 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2023-10-14T11:58:55.586 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2023-10-14T11:58:55.586 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-14T11:58:55.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-14T11:58:55.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-14T11:58:55.587 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.5", 2023-10-14T11:58:55.587 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQA5giplcTEKJhAAN7q3yytrGhq1rdhRohk+4A==", 2023-10-14T11:58:55.587 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2023-10-14T11:58:55.587 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2023-10-14T11:58:55.587 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2023-10-14T11:58:55.588 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2023-10-14T11:58:55.588 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-14T11:58:55.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-14T11:58:55.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-14T11:58:55.588 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.6", 2023-10-14T11:58:55.589 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQA8gipl7JU8JRAA0V4vMatkHCoyDp6W1kfB/w==", 2023-10-14T11:58:55.589 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2023-10-14T11:58:55.589 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2023-10-14T11:58:55.589 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2023-10-14T11:58:55.589 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2023-10-14T11:58:55.590 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-14T11:58:55.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-14T11:58:55.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-14T11:58:55.590 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.7", 2023-10-14T11:58:55.590 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQA/giplFXfHJBAAafwt2EliCJoj1qSq5utNcQ==", 2023-10-14T11:58:55.590 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2023-10-14T11:58:55.591 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2023-10-14T11:58:55.591 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2023-10-14T11:58:55.591 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2023-10-14T11:58:55.591 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-14T11:58:55.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-14T11:58:55.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-14T11:58:55.592 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.0", 2023-10-14T11:58:55.592 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQAmgiplorThEhAAXHGu+BK6WV8zJzecocBP5Q==", 2023-10-14T11:58:55.592 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2023-10-14T11:58:55.592 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2023-10-14T11:58:55.593 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow r", 2023-10-14T11:58:55.593 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow rw", 2023-10-14T11:58:55.593 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow rwx" 2023-10-14T11:58:55.593 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-14T11:58:55.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-14T11:58:55.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-14T11:58:55.594 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.1", 2023-10-14T11:58:55.594 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQAmgipl3JM0FhAA2tQCm3Ps6ItAbapH+C46dg==", 2023-10-14T11:58:55.594 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2023-10-14T11:58:55.594 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2023-10-14T11:58:55.594 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow r", 2023-10-14T11:58:55.595 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow rw", 2023-10-14T11:58:55.595 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow rwx" 2023-10-14T11:58:55.595 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-14T11:58:55.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-14T11:58:55.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-14T11:58:55.596 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.admin", 2023-10-14T11:58:55.596 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQAlgiplKJg4FhAAEkaZgSw/dlAk039MC9UFOg==", 2023-10-14T11:58:55.596 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2023-10-14T11:58:55.596 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2023-10-14T11:58:55.596 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2023-10-14T11:58:55.596 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2023-10-14T11:58:55.597 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2023-10-14T11:58:55.597 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-14T11:58:55.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-14T11:58:55.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-14T11:58:55.597 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.bootstrap-mds", 2023-10-14T11:58:55.598 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQBFgiplSVi1FRAAYvzlDq+8gmqTTLsC/nd7eQ==", 2023-10-14T11:58:55.598 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2023-10-14T11:58:55.598 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile bootstrap-mds" 2023-10-14T11:58:55.598 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-14T11:58:55.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-14T11:58:55.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-14T11:58:55.599 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.bootstrap-mgr", 2023-10-14T11:58:55.599 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQBFgiplQmG1FRAAldqtFiPEuH000v9yCF9qBw==", 2023-10-14T11:58:55.599 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2023-10-14T11:58:55.599 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile bootstrap-mgr" 2023-10-14T11:58:55.599 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-14T11:58:55.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-14T11:58:55.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-14T11:58:55.600 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.bootstrap-osd", 2023-10-14T11:58:55.600 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQBFgiplzGi1FRAAp+VJrfEE+gtCRC1IcEMdoQ==", 2023-10-14T11:58:55.600 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2023-10-14T11:58:55.601 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile bootstrap-osd" 2023-10-14T11:58:55.601 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-14T11:58:55.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-14T11:58:55.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-14T11:58:55.601 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.bootstrap-rbd", 2023-10-14T11:58:55.601 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQBFgiplb3G1FRAANtEa7Ca9oi0oSwjifY0Dbg==", 2023-10-14T11:58:55.602 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2023-10-14T11:58:55.602 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile bootstrap-rbd" 2023-10-14T11:58:55.602 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-14T11:58:55.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-14T11:58:55.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-14T11:58:55.603 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.bootstrap-rbd-mirror", 2023-10-14T11:58:55.603 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQBFgiplJHm1FRAABWY1UZCUax7/NOfDQsnYkw==", 2023-10-14T11:58:55.603 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2023-10-14T11:58:55.603 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2023-10-14T11:58:55.603 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-14T11:58:55.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-14T11:58:55.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-14T11:58:55.604 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.bootstrap-rgw", 2023-10-14T11:58:55.604 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQBFgipl3IC1FRAAPPKG1O8EMAvABbNkToW+lA==", 2023-10-14T11:58:55.604 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2023-10-14T11:58:55.604 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile bootstrap-rgw" 2023-10-14T11:58:55.605 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-14T11:58:55.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-14T11:58:55.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-14T11:58:55.605 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mgr.x", 2023-10-14T11:58:55.605 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQAlgiplxL7YMhAAlbcqrnh1wQDuNKCP2XLvFQ==", 2023-10-14T11:58:55.606 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2023-10-14T11:58:55.606 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2023-10-14T11:58:55.606 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile mgr", 2023-10-14T11:58:55.606 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2023-10-14T11:58:55.606 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-14T11:58:55.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-14T11:58:55.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-14T11:58:55.607 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mgr.y", 2023-10-14T11:58:55.607 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQAlgiplUokaLBAAJCZkFm8ccnPzzWOU3olSzw==", 2023-10-14T11:58:55.607 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2023-10-14T11:58:55.607 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2023-10-14T11:58:55.608 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile mgr", 2023-10-14T11:58:55.608 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2023-10-14T11:58:55.608 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-14T11:58:55.608 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-14T11:58:55.608 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2023-10-14T11:58:55.609 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-14T11:58:55.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.570+0000 7f63bcfb4700 1 -- 172.21.15.78:0/565339513 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f639806af00 msgr2=0x7f639806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:55.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.570+0000 7f63bcfb4700 1 --2- 172.21.15.78:0/565339513 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f639806af00 0x7f639806d3c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f63ac007d50 tx=0x7f63ac007000 comp rx=0 tx=0).stop 2023-10-14T11:58:55.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.570+0000 7f63bcfb4700 1 -- 172.21.15.78:0/565339513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63b806ba90 msgr2=0x7f63b806d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:55.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.570+0000 7f63bcfb4700 1 --2- 172.21.15.78:0/565339513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63b806ba90 0x7f63b806d340 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f63a8010840 tx=0x7f63a8010c00 comp rx=0 tx=0).stop 2023-10-14T11:58:55.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.570+0000 7f63bcfb4700 1 -- 172.21.15.78:0/565339513 shutdown_connections 2023-10-14T11:58:55.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.570+0000 7f63bcfb4700 1 --2- 172.21.15.78:0/565339513 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f639806af00 0x7f639806d3c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:55.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.570+0000 7f63bcfb4700 1 --2- 172.21.15.78:0/565339513 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63b8154bd0 0x7f63b8155080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:55.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.570+0000 7f63bcfb4700 1 --2- 172.21.15.78:0/565339513 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f63a0003a00 0x7f63b80a65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:55.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.570+0000 7f63bcfb4700 1 --2- 172.21.15.78:0/565339513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63b806ba90 0x7f63b806d340 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:55.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.570+0000 7f63bcfb4700 1 -- 172.21.15.78:0/565339513 >> 172.21.15.78:0/565339513 conn(0x7f63b80737d0 msgr2=0x7f63b80c36b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:55.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.570+0000 7f63bcfb4700 1 -- 172.21.15.78:0/565339513 shutdown_connections 2023-10-14T11:58:55.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.570+0000 7f63bcfb4700 1 -- 172.21.15.78:0/565339513 wait complete. 2023-10-14T11:58:55.613 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2023-10-14T11:58:55.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.733+0000 7eff18d21700 1 Processor -- start 2023-10-14T11:58:55.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.733+0000 7eff18d21700 1 -- start start 2023-10-14T11:58:55.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.734+0000 7eff18d21700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7eff1414c880 0x7eff14153080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:55.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.734+0000 7eff18d21700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7eff140c18c0 0x7eff1406f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:55.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.734+0000 7eff18d21700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7eff1406f950 0x7eff14156110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:55.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.734+0000 7eff18d21700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff14155c80 con 0x7eff1414c880 2023-10-14T11:58:55.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.734+0000 7eff18d21700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff140c46e0 con 0x7eff1406f950 2023-10-14T11:58:55.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.734+0000 7eff18d21700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff141535c0 con 0x7eff140c18c0 2023-10-14T11:58:55.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.734+0000 7eff1259c700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7eff1414c880 0x7eff14153080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:55.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.734+0000 7eff1259c700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7eff1414c880 0x7eff14153080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46026/0 (socket says 172.21.15.78:46026) 2023-10-14T11:58:55.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.734+0000 7eff1259c700 1 -- 172.21.15.78:0/1931021902 learned_addr learned my addr 172.21.15.78:0/1931021902 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:55.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.735+0000 7eff12d9d700 1 --2- 172.21.15.78:0/1931021902 >> v2:172.21.15.195:3300/0 conn(0x7eff1406f950 0x7eff14156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:55.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.735+0000 7eff11d9b700 1 --2- 172.21.15.78:0/1931021902 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7eff140c18c0 0x7eff1406f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:55.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.735+0000 7eff1259c700 1 -- 172.21.15.78:0/1931021902 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7eff140c18c0 msgr2=0x7eff1406f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:55.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.735+0000 7eff1259c700 1 --2- 172.21.15.78:0/1931021902 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7eff140c18c0 0x7eff1406f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:55.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.735+0000 7eff1259c700 1 -- 172.21.15.78:0/1931021902 >> v2:172.21.15.195:3300/0 conn(0x7eff1406f950 msgr2=0x7eff14156110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:55.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.735+0000 7eff1259c700 1 --2- 172.21.15.78:0/1931021902 >> v2:172.21.15.195:3300/0 conn(0x7eff1406f950 0x7eff14156110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:55.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.735+0000 7eff1259c700 1 -- 172.21.15.78:0/1931021902 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff140a8530 con 0x7eff1414c880 2023-10-14T11:58:55.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.735+0000 7eff11d9b700 1 --2- 172.21.15.78:0/1931021902 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7eff140c18c0 0x7eff1406f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:55.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.735+0000 7eff1259c700 1 --2- 172.21.15.78:0/1931021902 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff1414c880 0x7eff14153080 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7eff14071090 tx=0x7eff08009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4bcd96721d9f77a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:55.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.735+0000 7eff1159a700 1 -- 172.21.15.78:0/1931021902 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff0800b870 con 0x7eff1414c880 2023-10-14T11:58:55.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.735+0000 7eff1159a700 1 -- 172.21.15.78:0/1931021902 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff0800b9d0 con 0x7eff1414c880 2023-10-14T11:58:55.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.735+0000 7eff1159a700 1 -- 172.21.15.78:0/1931021902 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff0800bd60 con 0x7eff1414c880 2023-10-14T11:58:55.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.736+0000 7eff18d21700 1 -- 172.21.15.78:0/1931021902 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff1414c880 msgr2=0x7eff14153080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:55.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.736+0000 7eff18d21700 1 --2- 172.21.15.78:0/1931021902 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff1414c880 0x7eff14153080 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7eff14071090 tx=0x7eff08009bd0 comp rx=0 tx=0).stop 2023-10-14T11:58:55.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.736+0000 7eff18d21700 1 -- 172.21.15.78:0/1931021902 shutdown_connections 2023-10-14T11:58:55.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.736+0000 7eff18d21700 1 --2- 172.21.15.78:0/1931021902 >> v2:172.21.15.195:3300/0 conn(0x7eff1406f950 0x7eff14156110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:55.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.736+0000 7eff18d21700 1 --2- 172.21.15.78:0/1931021902 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff1414c880 0x7eff14153080 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:55.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.736+0000 7eff18d21700 1 --2- 172.21.15.78:0/1931021902 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7eff140c18c0 0x7eff1406f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:55.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.736+0000 7eff18d21700 1 -- 172.21.15.78:0/1931021902 >> 172.21.15.78:0/1931021902 conn(0x7eff140c39c0 msgr2=0x7eff140c3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:55.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.736+0000 7eff18d21700 1 -- 172.21.15.78:0/1931021902 shutdown_connections 2023-10-14T11:58:55.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.736+0000 7eff18d21700 1 -- 172.21.15.78:0/1931021902 wait complete. 2023-10-14T11:58:55.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.737+0000 7eff18d21700 1 Processor -- start 2023-10-14T11:58:55.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.737+0000 7eff18d21700 1 -- start start 2023-10-14T11:58:55.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.738+0000 7eff18d21700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7eff1406f950 0x7eff141eabb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:55.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.738+0000 7eff18d21700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7eff140c18c0 0x7eff140c04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:55.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.738+0000 7eff18d21700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff1414c880 0x7eff1406ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:55.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.738+0000 7eff18d21700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff14150230 con 0x7eff1414c880 2023-10-14T11:58:55.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.738+0000 7eff18d21700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff140c6be0 con 0x7eff1406f950 2023-10-14T11:58:55.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.738+0000 7eff18d21700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff141464a0 con 0x7eff140c18c0 2023-10-14T11:58:55.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.738+0000 7eff12d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff1414c880 0x7eff1406ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:55.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.738+0000 7eff12d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff1414c880 0x7eff1406ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46032/0 (socket says 172.21.15.78:46032) 2023-10-14T11:58:55.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.738+0000 7eff12d9d700 1 -- 172.21.15.78:0/246353611 learned_addr learned my addr 172.21.15.78:0/246353611 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:55.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.738+0000 7eff11d9b700 1 --2- 172.21.15.78:0/246353611 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7eff140c18c0 0x7eff140c04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:55.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.738+0000 7eff1259c700 1 --2- 172.21.15.78:0/246353611 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7eff1406f950 0x7eff141eabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:55.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.738+0000 7eff12d9d700 1 -- 172.21.15.78:0/246353611 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7eff140c18c0 msgr2=0x7eff140c04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:55.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.738+0000 7eff12d9d700 1 --2- 172.21.15.78:0/246353611 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7eff140c18c0 0x7eff140c04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:55.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.738+0000 7eff12d9d700 1 -- 172.21.15.78:0/246353611 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7eff1406f950 msgr2=0x7eff141eabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:55.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.738+0000 7eff12d9d700 1 --2- 172.21.15.78:0/246353611 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7eff1406f950 0x7eff141eabb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:55.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.738+0000 7eff12d9d700 1 -- 172.21.15.78:0/246353611 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff08009550 con 0x7eff1414c880 2023-10-14T11:58:55.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.738+0000 7eff12d9d700 1 --2- 172.21.15.78:0/246353611 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff1414c880 0x7eff1406ea70 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7eff0c007e60 tx=0x7eff0c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:55.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.738+0000 7eff02ffd700 1 -- 172.21.15.78:0/246353611 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff0c00ede0 con 0x7eff1414c880 2023-10-14T11:58:55.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.738+0000 7eff02ffd700 1 -- 172.21.15.78:0/246353611 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff0c014420 con 0x7eff1414c880 2023-10-14T11:58:55.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.738+0000 7eff18d21700 1 -- 172.21.15.78:0/246353611 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff141539f0 con 0x7eff1414c880 2023-10-14T11:58:55.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.738+0000 7eff02ffd700 1 -- 172.21.15.78:0/246353611 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff0c0147d0 con 0x7eff1414c880 2023-10-14T11:58:55.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.738+0000 7eff18d21700 1 -- 172.21.15.78:0/246353611 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff14153c40 con 0x7eff1414c880 2023-10-14T11:58:55.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.740+0000 7eff18d21700 1 -- 172.21.15.78:0/246353611 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff1414ff90 con 0x7eff1414c880 2023-10-14T11:58:55.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.740+0000 7eff02ffd700 1 -- 172.21.15.78:0/246353611 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7eff0c014930 con 0x7eff1414c880 2023-10-14T11:58:55.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.740+0000 7eff02ffd700 1 --2- 172.21.15.78:0/246353611 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7efef406af00 0x7efef406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:55.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.740+0000 7eff02ffd700 1 -- 172.21.15.78:0/246353611 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 4857+0+0 (secure 0 0 0) 0x7eff0c0899a0 con 0x7eff1414c880 2023-10-14T11:58:55.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.741+0000 7eff1259c700 1 --2- 172.21.15.78:0/246353611 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7efef406af00 0x7efef406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:55.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.741+0000 7eff1259c700 1 --2- 172.21.15.78:0/246353611 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7efef406af00 0x7efef406d3c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7eff08014040 tx=0x7eff08008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:55.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:55.745+0000 7eff02ffd700 1 -- 172.21.15.78:0/246353611 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7eff1414ff90 con 0x7eff1414c880 2023-10-14T11:58:56.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.012+0000 7eff18d21700 1 -- 172.21.15.78:0/246353611 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7eff14150370 con 0x7eff1414c880 2023-10-14T11:58:56.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.014+0000 7eff02ffd700 1 -- 172.21.15.78:0/246353611 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v9) v1 ==== 90+0+0 (secure 0 0 0) 0x7eff14150370 con 0x7eff1414c880 2023-10-14T11:58:56.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.016+0000 7eff18d21700 1 -- 172.21.15.78:0/246353611 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7efef406af00 msgr2=0x7efef406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:56.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.016+0000 7eff18d21700 1 --2- 172.21.15.78:0/246353611 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7efef406af00 0x7efef406d3c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7eff08014040 tx=0x7eff08008030 comp rx=0 tx=0).stop 2023-10-14T11:58:56.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.016+0000 7eff18d21700 1 -- 172.21.15.78:0/246353611 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff1414c880 msgr2=0x7eff1406ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:56.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.016+0000 7eff18d21700 1 --2- 172.21.15.78:0/246353611 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff1414c880 0x7eff1406ea70 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7eff0c007e60 tx=0x7eff0c00e3c0 comp rx=0 tx=0).stop 2023-10-14T11:58:56.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.016+0000 7eff18d21700 1 -- 172.21.15.78:0/246353611 shutdown_connections 2023-10-14T11:58:56.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.016+0000 7eff18d21700 1 --2- 172.21.15.78:0/246353611 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff1414c880 0x7eff1406ea70 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:56.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.016+0000 7eff18d21700 1 --2- 172.21.15.78:0/246353611 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7efef406af00 0x7efef406d3c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:56.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.016+0000 7eff18d21700 1 --2- 172.21.15.78:0/246353611 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7eff1406f950 0x7eff141eabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:56.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.016+0000 7eff18d21700 1 --2- 172.21.15.78:0/246353611 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7eff140c18c0 0x7eff140c04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:56.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.016+0000 7eff18d21700 1 -- 172.21.15.78:0/246353611 >> 172.21.15.78:0/246353611 conn(0x7eff140c39c0 msgr2=0x7eff1406a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:56.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.016+0000 7eff18d21700 1 -- 172.21.15.78:0/246353611 shutdown_connections 2023-10-14T11:58:56.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.016+0000 7eff18d21700 1 -- 172.21.15.78:0/246353611 wait complete. 2023-10-14T11:58:56.029 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2023-10-14T11:58:56.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.152+0000 7f33eb5f9700 1 Processor -- start 2023-10-14T11:58:56.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.152+0000 7f33eb5f9700 1 -- start start 2023-10-14T11:58:56.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.153+0000 7f33eb5f9700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f33e406f430 0x7f33e406f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:56.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.153+0000 7f33eb5f9700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f33e4155e00 0x7f33e41562b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:56.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.153+0000 7f33eb5f9700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33e4152850 con 0x7f33e4072390 2023-10-14T11:58:56.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.153+0000 7f33eb5f9700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33e413bf20 con 0x7f33e406f430 2023-10-14T11:58:56.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.153+0000 7f33eb5f9700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33e4152b10 con 0x7f33e4155e00 2023-10-14T11:58:56.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.153+0000 7f33e9395700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f33e406f430 0x7f33e406f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:56.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.153+0000 7f33e8b94700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f33e4072390 0x7f33e413b810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50918/0 (socket says 172.21.15.78:50918) 2023-10-14T11:58:56.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.153+0000 7f33e8b94700 1 -- 172.21.15.78:0/4285610482 learned_addr learned my addr 172.21.15.78:0/4285610482 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:56.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.153+0000 7f33e9b96700 1 --2- 172.21.15.78:0/4285610482 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f33e4155e00 0x7f33e41562b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:56.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.153+0000 7f33e9395700 1 -- 172.21.15.78:0/4285610482 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f33e4155e00 msgr2=0x7f33e41562b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:56.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.153+0000 7f33e9395700 1 --2- 172.21.15.78:0/4285610482 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f33e4155e00 0x7f33e41562b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:56.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.153+0000 7f33e9395700 1 -- 172.21.15.78:0/4285610482 >> v1:172.21.15.78:6789/0 conn(0x7f33e4072390 legacy=0x7f33e413b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:56.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.153+0000 7f33e9395700 1 -- 172.21.15.78:0/4285610482 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e4152df0 con 0x7f33e406f430 2023-10-14T11:58:56.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.153+0000 7f33dbfff700 1 -- 172.21.15.78:0/4285610482 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1042812166 0 0) 0x7f33e4152850 con 0x7f33e4072390 2023-10-14T11:58:56.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.153+0000 7f33e9395700 1 --2- 172.21.15.78:0/4285610482 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33e406f430 0x7f33e406f810 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f33e0009840 tx=0x7f33e0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a599e2339d3e7471 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:56.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.153+0000 7f33dbfff700 1 -- 172.21.15.78:0/4285610482 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e0011830 con 0x7f33e406f430 2023-10-14T11:58:56.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.153+0000 7f33dbfff700 1 -- 172.21.15.78:0/4285610482 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33e0011990 con 0x7f33e406f430 2023-10-14T11:58:56.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.153+0000 7f33dbfff700 1 -- 172.21.15.78:0/4285610482 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e0011d20 con 0x7f33e406f430 2023-10-14T11:58:56.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.154+0000 7f33eb5f9700 1 -- 172.21.15.78:0/4285610482 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33e406f430 msgr2=0x7f33e406f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:56.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.154+0000 7f33eb5f9700 1 --2- 172.21.15.78:0/4285610482 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33e406f430 0x7f33e406f810 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f33e0009840 tx=0x7f33e0009b50 comp rx=0 tx=0).stop 2023-10-14T11:58:56.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.154+0000 7f33eb5f9700 1 -- 172.21.15.78:0/4285610482 shutdown_connections 2023-10-14T11:58:56.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.154+0000 7f33eb5f9700 1 --2- 172.21.15.78:0/4285610482 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f33e4155e00 0x7f33e41562b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:56.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.154+0000 7f33eb5f9700 1 --2- 172.21.15.78:0/4285610482 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33e406f430 0x7f33e406f810 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:56.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.154+0000 7f33eb5f9700 1 -- 172.21.15.78:0/4285610482 >> 172.21.15.78:0/4285610482 conn(0x7f33e40c3a50 msgr2=0x7f33e40c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:56.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.154+0000 7f33eb5f9700 1 -- 172.21.15.78:0/4285610482 shutdown_connections 2023-10-14T11:58:56.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.154+0000 7f33eb5f9700 1 -- 172.21.15.78:0/4285610482 wait complete. 2023-10-14T11:58:56.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.155+0000 7f33eb5f9700 1 Processor -- start 2023-10-14T11:58:56.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.156+0000 7f33eb5f9700 1 -- start start 2023-10-14T11:58:56.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.156+0000 7f33eb5f9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33e406f430 0x7f33e4132f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:56.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.156+0000 7f33eb5f9700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f33e4072390 0x7f33e41545c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:56.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.156+0000 7f33eb5f9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f33e4155e00 0x7f33e40bff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:56.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.156+0000 7f33eb5f9700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e40bce00 con 0x7f33e406f430 2023-10-14T11:58:56.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.156+0000 7f33eb5f9700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e41559b0 con 0x7f33e4155e00 2023-10-14T11:58:56.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.156+0000 7f33eb5f9700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33e41e90d0 con 0x7f33e4072390 2023-10-14T11:58:56.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.156+0000 7f33e9395700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33e406f430 0x7f33e4132f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:56.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.157+0000 7f33e9395700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33e406f430 0x7f33e4132f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46046/0 (socket says 172.21.15.78:46046) 2023-10-14T11:58:56.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.157+0000 7f33e9395700 1 -- 172.21.15.78:0/19711903 learned_addr learned my addr 172.21.15.78:0/19711903 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:56.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.157+0000 7f33e8b94700 1 --2- 172.21.15.78:0/19711903 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f33e4072390 0x7f33e41545c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:56.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.157+0000 7f33e9395700 1 -- 172.21.15.78:0/19711903 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f33e4072390 msgr2=0x7f33e41545c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:56.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.157+0000 7f33e9395700 1 --2- 172.21.15.78:0/19711903 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f33e4072390 0x7f33e41545c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:56.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.157+0000 7f33e9395700 1 -- 172.21.15.78:0/19711903 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f33e4155e00 msgr2=0x7f33e40bff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:56.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.157+0000 7f33e9395700 1 --2- 172.21.15.78:0/19711903 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f33e4155e00 0x7f33e40bff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:56.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.157+0000 7f33e9395700 1 -- 172.21.15.78:0/19711903 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e00094a0 con 0x7f33e406f430 2023-10-14T11:58:56.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.157+0000 7f33e9395700 1 --2- 172.21.15.78:0/19711903 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33e406f430 0x7f33e4132f70 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f33e0009b90 tx=0x7f33e0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:56.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.157+0000 7f33d9ffb700 1 -- 172.21.15.78:0/19711903 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e00036a0 con 0x7f33e406f430 2023-10-14T11:58:56.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.157+0000 7f33d9ffb700 1 -- 172.21.15.78:0/19711903 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33e001f070 con 0x7f33e406f430 2023-10-14T11:58:56.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.157+0000 7f33eb5f9700 1 -- 172.21.15.78:0/19711903 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33e41e8dd0 con 0x7f33e406f430 2023-10-14T11:58:56.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.157+0000 7f33d9ffb700 1 -- 172.21.15.78:0/19711903 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e0003a40 con 0x7f33e406f430 2023-10-14T11:58:56.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.157+0000 7f33eb5f9700 1 -- 172.21.15.78:0/19711903 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33e40bc2e0 con 0x7f33e406f430 2023-10-14T11:58:56.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.158+0000 7f33eb5f9700 1 -- 172.21.15.78:0/19711903 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33e4074b80 con 0x7f33e406f430 2023-10-14T11:58:56.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.159+0000 7f33d9ffb700 1 -- 172.21.15.78:0/19711903 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f33e0003ba0 con 0x7f33e406f430 2023-10-14T11:58:56.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.159+0000 7f33d9ffb700 1 --2- 172.21.15.78:0/19711903 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f33c806af30 0x7f33c806d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:56.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.159+0000 7f33d9ffb700 1 -- 172.21.15.78:0/19711903 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f33e0021070 con 0x7f33e406f430 2023-10-14T11:58:56.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.159+0000 7f33e8b94700 1 --2- 172.21.15.78:0/19711903 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f33c806af30 0x7f33c806d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:56.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.159+0000 7f33e8b94700 1 --2- 172.21.15.78:0/19711903 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f33c806af30 0x7f33c806d3f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f33e41ead90 tx=0x7f33d4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:56.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.164+0000 7f33d9ffb700 1 -- 172.21.15.78:0/19711903 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f33e00560d0 con 0x7f33e406f430 2023-10-14T11:58:56.430 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.429+0000 7f33eb5f9700 1 -- 172.21.15.78:0/19711903 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f33e40c2390 con 0x7f33e406f430 2023-10-14T11:58:56.431 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.431+0000 7f33d9ffb700 1 -- 172.21.15.78:0/19711903 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7f33e00150a0 con 0x7f33e406f430 2023-10-14T11:58:56.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.433+0000 7f33eb5f9700 1 -- 172.21.15.78:0/19711903 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f33c806af30 msgr2=0x7f33c806d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:56.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.433+0000 7f33eb5f9700 1 --2- 172.21.15.78:0/19711903 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f33c806af30 0x7f33c806d3f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f33e41ead90 tx=0x7f33d4009040 comp rx=0 tx=0).stop 2023-10-14T11:58:56.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.434+0000 7f33eb5f9700 1 -- 172.21.15.78:0/19711903 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33e406f430 msgr2=0x7f33e4132f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:56.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.434+0000 7f33eb5f9700 1 --2- 172.21.15.78:0/19711903 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33e406f430 0x7f33e4132f70 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f33e0009b90 tx=0x7f33e0004750 comp rx=0 tx=0).stop 2023-10-14T11:58:56.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.434+0000 7f33eb5f9700 1 -- 172.21.15.78:0/19711903 shutdown_connections 2023-10-14T11:58:56.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.434+0000 7f33eb5f9700 1 --2- 172.21.15.78:0/19711903 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f33c806af30 0x7f33c806d3f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:56.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.434+0000 7f33eb5f9700 1 --2- 172.21.15.78:0/19711903 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f33e4155e00 0x7f33e40bff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:56.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.434+0000 7f33eb5f9700 1 --2- 172.21.15.78:0/19711903 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33e406f430 0x7f33e4132f70 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:56.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.434+0000 7f33eb5f9700 1 --2- 172.21.15.78:0/19711903 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f33e4072390 0x7f33e41545c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:56.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.434+0000 7f33eb5f9700 1 -- 172.21.15.78:0/19711903 >> 172.21.15.78:0/19711903 conn(0x7f33e40c3a50 msgr2=0x7f33e40a8490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:56.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.434+0000 7f33eb5f9700 1 -- 172.21.15.78:0/19711903 shutdown_connections 2023-10-14T11:58:56.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.434+0000 7f33eb5f9700 1 -- 172.21.15.78:0/19711903 wait complete. 2023-10-14T11:58:56.446 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2023-10-14T11:58:56.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.567+0000 7fc3fe05b700 1 Processor -- start 2023-10-14T11:58:56.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.567+0000 7fc3fe05b700 1 -- start start 2023-10-14T11:58:56.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.568+0000 7fc3fe05b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fc3f80c4f40 0x7fc3f80c66e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:56.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.568+0000 7fc3fe05b700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc3f806b180 0x7fc3f806b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:56.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.568+0000 7fc3fe05b700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f8155ab0 con 0x7fc3f80afad0 2023-10-14T11:58:56.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.568+0000 7fc3fe05b700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3f8156680 con 0x7fc3f806b180 2023-10-14T11:58:56.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.568+0000 7fc3fe05b700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3f80c6250 con 0x7fc3f80c4f40 2023-10-14T11:58:56.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.568+0000 7fc3f7fff700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc3f806b180 0x7fc3f806b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:56.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.568+0000 7fc3f77fe700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc3f80afad0 0x7fc3f8155f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50930/0 (socket says 172.21.15.78:50930) 2023-10-14T11:58:56.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.568+0000 7fc3f77fe700 1 -- 172.21.15.78:0/514696549 learned_addr learned my addr 172.21.15.78:0/514696549 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:56.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.568+0000 7fc3f7fff700 1 -- 172.21.15.78:0/514696549 >> v2:172.21.15.195:3300/0 conn(0x7fc3f80c4f40 msgr2=0x7fc3f80c66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:56.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.568+0000 7fc3f6ffd700 1 --2- 172.21.15.78:0/514696549 >> v2:172.21.15.195:3300/0 conn(0x7fc3f80c4f40 0x7fc3f80c66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:56.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.568+0000 7fc3f67fc700 1 -- 172.21.15.78:0/514696549 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 257816668 0 0) 0x7fc3f8155ab0 con 0x7fc3f80afad0 2023-10-14T11:58:56.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.568+0000 7fc3f7fff700 1 --2- 172.21.15.78:0/514696549 >> v2:172.21.15.195:3300/0 conn(0x7fc3f80c4f40 0x7fc3f80c66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:56.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.569+0000 7fc3f7fff700 1 -- 172.21.15.78:0/514696549 >> v1:172.21.15.78:6789/0 conn(0x7fc3f80afad0 legacy=0x7fc3f8155f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:56.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.569+0000 7fc3f7fff700 1 -- 172.21.15.78:0/514696549 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f81525a0 con 0x7fc3f806b180 2023-10-14T11:58:56.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.569+0000 7fc3f7fff700 1 --2- 172.21.15.78:0/514696549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3f806b180 0x7fc3f806b590 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc3f806b050 tx=0x7fc3f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1222885ce9c49e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:56.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.569+0000 7fc3f67fc700 1 -- 172.21.15.78:0/514696549 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f000e070 con 0x7fc3f806b180 2023-10-14T11:58:56.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.569+0000 7fc3f67fc700 1 -- 172.21.15.78:0/514696549 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f000bbd0 con 0x7fc3f806b180 2023-10-14T11:58:56.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.569+0000 7fc3f67fc700 1 -- 172.21.15.78:0/514696549 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f0012430 con 0x7fc3f806b180 2023-10-14T11:58:56.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.569+0000 7fc3fe05b700 1 -- 172.21.15.78:0/514696549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3f806b180 msgr2=0x7fc3f806b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:56.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.569+0000 7fc3fe05b700 1 --2- 172.21.15.78:0/514696549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3f806b180 0x7fc3f806b590 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc3f806b050 tx=0x7fc3f000b330 comp rx=0 tx=0).stop 2023-10-14T11:58:56.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.569+0000 7fc3fe05b700 1 -- 172.21.15.78:0/514696549 shutdown_connections 2023-10-14T11:58:56.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.569+0000 7fc3fe05b700 1 --2- 172.21.15.78:0/514696549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3f806b180 0x7fc3f806b590 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:56.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.569+0000 7fc3fe05b700 1 --2- 172.21.15.78:0/514696549 >> v2:172.21.15.195:3300/0 conn(0x7fc3f80c4f40 0x7fc3f80c66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:56.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.569+0000 7fc3fe05b700 1 -- 172.21.15.78:0/514696549 >> 172.21.15.78:0/514696549 conn(0x7fc3f80c6e20 msgr2=0x7fc3f80c7200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:56.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.569+0000 7fc3fe05b700 1 -- 172.21.15.78:0/514696549 shutdown_connections 2023-10-14T11:58:56.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.569+0000 7fc3fe05b700 1 -- 172.21.15.78:0/514696549 wait complete. 2023-10-14T11:58:56.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.571+0000 7fc3fe05b700 1 Processor -- start 2023-10-14T11:58:56.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.571+0000 7fc3fe05b700 1 -- start start 2023-10-14T11:58:56.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.571+0000 7fc3fe05b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3f806b180 0x7fc3f8157880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:56.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.571+0000 7fc3fe05b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3f80afad0 0x7fc3f814e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:56.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.571+0000 7fc3fe05b700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc3f80c4f40 0x7fc3f80aa0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:56.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.571+0000 7fc3fe05b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f80a9c50 con 0x7fc3f80afad0 2023-10-14T11:58:56.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.571+0000 7fc3fe05b700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f80aba00 con 0x7fc3f806b180 2023-10-14T11:58:56.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.571+0000 7fc3fe05b700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f80abc70 con 0x7fc3f80c4f40 2023-10-14T11:58:56.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.572+0000 7fc3f6ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3f80afad0 0x7fc3f814e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:56.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.572+0000 7fc3f6ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3f80afad0 0x7fc3f814e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46062/0 (socket says 172.21.15.78:46062) 2023-10-14T11:58:56.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.572+0000 7fc3f6ffd700 1 -- 172.21.15.78:0/4116613463 learned_addr learned my addr 172.21.15.78:0/4116613463 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:56.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.572+0000 7fc3f77fe700 1 --2- 172.21.15.78:0/4116613463 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3f806b180 0x7fc3f8157880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:56.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.572+0000 7fc3f7fff700 1 --2- 172.21.15.78:0/4116613463 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc3f80c4f40 0x7fc3f80aa0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:56.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.572+0000 7fc3f6ffd700 1 -- 172.21.15.78:0/4116613463 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc3f80c4f40 msgr2=0x7fc3f80aa0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:56.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.572+0000 7fc3f6ffd700 1 --2- 172.21.15.78:0/4116613463 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc3f80c4f40 0x7fc3f80aa0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:56.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.572+0000 7fc3f6ffd700 1 -- 172.21.15.78:0/4116613463 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3f806b180 msgr2=0x7fc3f8157880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:56.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.572+0000 7fc3f6ffd700 1 --2- 172.21.15.78:0/4116613463 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3f806b180 0x7fc3f8157880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:56.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.572+0000 7fc3f6ffd700 1 -- 172.21.15.78:0/4116613463 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f0009a70 con 0x7fc3f80afad0 2023-10-14T11:58:56.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.572+0000 7fc3f7fff700 1 --2- 172.21.15.78:0/4116613463 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc3f80c4f40 0x7fc3f80aa0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:56.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.572+0000 7fc3f6ffd700 1 --2- 172.21.15.78:0/4116613463 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3f80afad0 0x7fc3f814e3d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc3e800c820 tx=0x7fc3e800cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:56.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.572+0000 7fc3dffff700 1 -- 172.21.15.78:0/4116613463 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e8011800 con 0x7fc3f80afad0 2023-10-14T11:58:56.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.572+0000 7fc3dffff700 1 -- 172.21.15.78:0/4116613463 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e8011960 con 0x7fc3f80afad0 2023-10-14T11:58:56.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.572+0000 7fc3dffff700 1 -- 172.21.15.78:0/4116613463 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e8011cf0 con 0x7fc3f80afad0 2023-10-14T11:58:56.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.572+0000 7fc3fe05b700 1 -- 172.21.15.78:0/4116613463 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3f80abf50 con 0x7fc3f80afad0 2023-10-14T11:58:56.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.572+0000 7fc3fe05b700 1 -- 172.21.15.78:0/4116613463 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3f81ea8c0 con 0x7fc3f80afad0 2023-10-14T11:58:56.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.573+0000 7fc3fe05b700 1 -- 172.21.15.78:0/4116613463 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3f80a9dc0 con 0x7fc3f80afad0 2023-10-14T11:58:56.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.573+0000 7fc3dffff700 1 -- 172.21.15.78:0/4116613463 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc3e8018360 con 0x7fc3f80afad0 2023-10-14T11:58:56.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.574+0000 7fc3dffff700 1 --2- 172.21.15.78:0/4116613463 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc3e406af00 0x7fc3e406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:56.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.574+0000 7fc3dffff700 1 -- 172.21.15.78:0/4116613463 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc3e8089070 con 0x7fc3f80afad0 2023-10-14T11:58:56.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.574+0000 7fc3f77fe700 1 --2- 172.21.15.78:0/4116613463 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc3e406af00 0x7fc3e406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:56.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.574+0000 7fc3f77fe700 1 --2- 172.21.15.78:0/4116613463 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc3e406af00 0x7fc3e406d3c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec007fd0 tx=0x7fc3ec007de0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:56.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.578+0000 7fc3dffff700 1 -- 172.21.15.78:0/4116613463 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc3f80a9dc0 con 0x7fc3f80afad0 2023-10-14T11:58:56.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.842+0000 7fc3fe05b700 1 -- 172.21.15.78:0/4116613463 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7fc3f80a9dc0 con 0x7fc3f80afad0 2023-10-14T11:58:56.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.844+0000 7fc3dffff700 1 -- 172.21.15.78:0/4116613463 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v11) v1 ==== 90+0+0 (secure 0 0 0) 0x7fc3f80a9dc0 con 0x7fc3f80afad0 2023-10-14T11:58:56.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.847+0000 7fc3fe05b700 1 -- 172.21.15.78:0/4116613463 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc3e406af00 msgr2=0x7fc3e406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:56.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.847+0000 7fc3fe05b700 1 --2- 172.21.15.78:0/4116613463 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc3e406af00 0x7fc3e406d3c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec007fd0 tx=0x7fc3ec007de0 comp rx=0 tx=0).stop 2023-10-14T11:58:56.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.847+0000 7fc3fe05b700 1 -- 172.21.15.78:0/4116613463 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3f80afad0 msgr2=0x7fc3f814e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:56.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.847+0000 7fc3fe05b700 1 --2- 172.21.15.78:0/4116613463 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3f80afad0 0x7fc3f814e3d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc3e800c820 tx=0x7fc3e800cb30 comp rx=0 tx=0).stop 2023-10-14T11:58:56.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.847+0000 7fc3fe05b700 1 -- 172.21.15.78:0/4116613463 shutdown_connections 2023-10-14T11:58:56.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.847+0000 7fc3fe05b700 1 --2- 172.21.15.78:0/4116613463 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc3f80c4f40 0x7fc3f80aa0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:56.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.847+0000 7fc3fe05b700 1 --2- 172.21.15.78:0/4116613463 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc3e406af00 0x7fc3e406d3c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:56.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.847+0000 7fc3fe05b700 1 --2- 172.21.15.78:0/4116613463 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3f806b180 0x7fc3f8157880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:56.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.847+0000 7fc3fe05b700 1 --2- 172.21.15.78:0/4116613463 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3f80afad0 0x7fc3f814e3d0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:56.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.847+0000 7fc3fe05b700 1 -- 172.21.15.78:0/4116613463 >> 172.21.15.78:0/4116613463 conn(0x7fc3f80c6e20 msgr2=0x7fc3f80ae980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:56.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.847+0000 7fc3fe05b700 1 -- 172.21.15.78:0/4116613463 shutdown_connections 2023-10-14T11:58:56.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.847+0000 7fc3fe05b700 1 -- 172.21.15.78:0/4116613463 wait complete. 2023-10-14T11:58:56.860 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2023-10-14T11:58:56.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.981+0000 7f8b8f382700 1 Processor -- start 2023-10-14T11:58:56.982 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.981+0000 7f8b8f382700 1 -- start start 2023-10-14T11:58:56.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.981+0000 7f8b8f382700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8b8813dbe0 0x7f8b8806a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:56.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.982+0000 7f8b8f382700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f8b8806ae10 0x7f8b880c09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:56.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.982+0000 7f8b8f382700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b8806ebe0 con 0x7f8b880c4540 2023-10-14T11:58:56.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.982+0000 7f8b8f382700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b8806d330 con 0x7f8b8813dbe0 2023-10-14T11:58:56.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.982+0000 7f8b8f382700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b881558c0 con 0x7f8b8806ae10 2023-10-14T11:58:56.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.982+0000 7f8b8d11e700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8b8813dbe0 0x7f8b8806a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:56.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.982+0000 7f8b8d11e700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8b8813dbe0 0x7f8b8806a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46068/0 (socket says 172.21.15.78:46068) 2023-10-14T11:58:56.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.982+0000 7f8b8d11e700 1 -- 172.21.15.78:0/938379039 learned_addr learned my addr 172.21.15.78:0/938379039 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:56.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.982+0000 7f8b8c91d700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f8b880c4540 0x7f8b880c02e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50944/0 (socket says 172.21.15.78:50944) 2023-10-14T11:58:56.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.982+0000 7f8b8d11e700 1 -- 172.21.15.78:0/938379039 >> v2:172.21.15.195:3300/0 conn(0x7f8b8806ae10 msgr2=0x7f8b880c09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:56.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.982+0000 7f8b8d11e700 1 --2- 172.21.15.78:0/938379039 >> v2:172.21.15.195:3300/0 conn(0x7f8b8806ae10 0x7f8b880c09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:56.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.982+0000 7f8b8d11e700 1 -- 172.21.15.78:0/938379039 >> v1:172.21.15.78:6789/0 conn(0x7f8b880c4540 legacy=0x7f8b880c02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:56.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.982+0000 7f8b8d11e700 1 -- 172.21.15.78:0/938379039 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b88154b00 con 0x7f8b8813dbe0 2023-10-14T11:58:56.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.982+0000 7f8b7bfff700 1 -- 172.21.15.78:0/938379039 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3276441191 0 0) 0x7f8b8806ebe0 con 0x7f8b880c4540 2023-10-14T11:58:56.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.982+0000 7f8b8d11e700 1 --2- 172.21.15.78:0/938379039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b8813dbe0 0x7f8b8806a8d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8b880c3a00 tx=0x7f8b84009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3182551d813b691c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:56.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.982+0000 7f8b7bfff700 1 -- 172.21.15.78:0/938379039 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8400b870 con 0x7f8b8813dbe0 2023-10-14T11:58:56.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.982+0000 7f8b7bfff700 1 -- 172.21.15.78:0/938379039 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b8400b9d0 con 0x7f8b8813dbe0 2023-10-14T11:58:56.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.982+0000 7f8b7bfff700 1 -- 172.21.15.78:0/938379039 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8400bd60 con 0x7f8b8813dbe0 2023-10-14T11:58:56.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.983+0000 7f8b8f382700 1 -- 172.21.15.78:0/938379039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b8813dbe0 msgr2=0x7f8b8806a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:56.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.983+0000 7f8b8f382700 1 --2- 172.21.15.78:0/938379039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b8813dbe0 0x7f8b8806a8d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8b880c3a00 tx=0x7f8b84009bd0 comp rx=0 tx=0).stop 2023-10-14T11:58:56.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.983+0000 7f8b8f382700 1 -- 172.21.15.78:0/938379039 shutdown_connections 2023-10-14T11:58:56.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.983+0000 7f8b8f382700 1 --2- 172.21.15.78:0/938379039 >> v2:172.21.15.195:3300/0 conn(0x7f8b8806ae10 0x7f8b880c09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:56.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.983+0000 7f8b8f382700 1 --2- 172.21.15.78:0/938379039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b8813dbe0 0x7f8b8806a8d0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:56.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.983+0000 7f8b8f382700 1 -- 172.21.15.78:0/938379039 >> 172.21.15.78:0/938379039 conn(0x7f8b880c3a50 msgr2=0x7f8b880c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:56.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.983+0000 7f8b8f382700 1 -- 172.21.15.78:0/938379039 shutdown_connections 2023-10-14T11:58:56.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.983+0000 7f8b8f382700 1 -- 172.21.15.78:0/938379039 wait complete. 2023-10-14T11:58:56.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.984+0000 7f8b8f382700 1 Processor -- start 2023-10-14T11:58:56.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.984+0000 7f8b8f382700 1 -- start start 2023-10-14T11:58:56.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.985+0000 7f8b8f382700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8b8806ae10 0x7f8b88138d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:56.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.985+0000 7f8b8f382700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b880c4540 0x7f8b880c25d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:56.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.985+0000 7f8b8f382700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8b8813dbe0 0x7f8b881ed870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:56.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.985+0000 7f8b8f382700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b88155d20 con 0x7f8b880c4540 2023-10-14T11:58:56.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.985+0000 7f8b8f382700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b88158630 con 0x7f8b8813dbe0 2023-10-14T11:58:56.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.985+0000 7f8b8f382700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b881eace0 con 0x7f8b8806ae10 2023-10-14T11:58:56.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.985+0000 7f8b8c91d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b880c4540 0x7f8b880c25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:56.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.985+0000 7f8b8c91d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b880c4540 0x7f8b880c25d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46076/0 (socket says 172.21.15.78:46076) 2023-10-14T11:58:56.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.985+0000 7f8b8c91d700 1 -- 172.21.15.78:0/2269702908 learned_addr learned my addr 172.21.15.78:0/2269702908 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:56.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.986+0000 7f8b8d11e700 1 --2- 172.21.15.78:0/2269702908 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8b8806ae10 0x7f8b88138d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:56.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.986+0000 7f8b8c91d700 1 -- 172.21.15.78:0/2269702908 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8b8806ae10 msgr2=0x7f8b88138d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:56.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.986+0000 7f8b8c91d700 1 --2- 172.21.15.78:0/2269702908 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8b8806ae10 0x7f8b88138d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:56.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.986+0000 7f8b8c91d700 1 -- 172.21.15.78:0/2269702908 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8b8813dbe0 msgr2=0x7f8b881ed870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:56.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.986+0000 7f8b8d91f700 1 --2- 172.21.15.78:0/2269702908 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8b8813dbe0 0x7f8b881ed870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:56.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.986+0000 7f8b8c91d700 1 --2- 172.21.15.78:0/2269702908 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8b8813dbe0 0x7f8b881ed870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:56.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.986+0000 7f8b8c91d700 1 -- 172.21.15.78:0/2269702908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b84009550 con 0x7f8b880c4540 2023-10-14T11:58:56.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.986+0000 7f8b8c91d700 1 --2- 172.21.15.78:0/2269702908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b880c4540 0x7f8b880c25d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c00c540 tx=0x7f8b7c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:56.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.986+0000 7f8b79ffb700 1 -- 172.21.15.78:0/2269702908 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7c004c80 con 0x7f8b880c4540 2023-10-14T11:58:56.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.986+0000 7f8b79ffb700 1 -- 172.21.15.78:0/2269702908 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b7c004de0 con 0x7f8b880c4540 2023-10-14T11:58:56.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.986+0000 7f8b8f382700 1 -- 172.21.15.78:0/2269702908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b88158880 con 0x7f8b880c4540 2023-10-14T11:58:56.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.986+0000 7f8b79ffb700 1 -- 172.21.15.78:0/2269702908 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7c007310 con 0x7f8b880c4540 2023-10-14T11:58:56.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.986+0000 7f8b8f382700 1 -- 172.21.15.78:0/2269702908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b88157dc0 con 0x7f8b880c4540 2023-10-14T11:58:56.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.987+0000 7f8b8f382700 1 -- 172.21.15.78:0/2269702908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b88074b80 con 0x7f8b880c4540 2023-10-14T11:58:56.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.987+0000 7f8b79ffb700 1 -- 172.21.15.78:0/2269702908 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8b7c007470 con 0x7f8b880c4540 2023-10-14T11:58:57.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.988+0000 7f8b79ffb700 1 --2- 172.21.15.78:0/2269702908 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8b6c06af00 0x7f8b6c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:57.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.988+0000 7f8b8d11e700 1 --2- 172.21.15.78:0/2269702908 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8b6c06af00 0x7f8b6c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:57.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.988+0000 7f8b79ffb700 1 -- 172.21.15.78:0/2269702908 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8b7c089010 con 0x7f8b880c4540 2023-10-14T11:58:57.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.988+0000 7f8b8d11e700 1 --2- 172.21.15.78:0/2269702908 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8b6c06af00 0x7f8b6c06d3c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8b84009c10 tx=0x7f8b84008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:57.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:56.992+0000 7f8b79ffb700 1 -- 172.21.15.78:0/2269702908 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8b7c054410 con 0x7f8b880c4540 2023-10-14T11:58:57.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.262+0000 7f8b8f382700 1 -- 172.21.15.78:0/2269702908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f8b8806b2e0 con 0x7f8b880c4540 2023-10-14T11:58:57.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.264+0000 7f8b79ffb700 1 -- 172.21.15.78:0/2269702908 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v12) v1 ==== 90+0+0 (secure 0 0 0) 0x7f8b7c057a30 con 0x7f8b880c4540 2023-10-14T11:58:57.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.267+0000 7f8b8f382700 1 -- 172.21.15.78:0/2269702908 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8b6c06af00 msgr2=0x7f8b6c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:57.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.267+0000 7f8b8f382700 1 --2- 172.21.15.78:0/2269702908 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8b6c06af00 0x7f8b6c06d3c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8b84009c10 tx=0x7f8b84008030 comp rx=0 tx=0).stop 2023-10-14T11:58:57.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.267+0000 7f8b8f382700 1 -- 172.21.15.78:0/2269702908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b880c4540 msgr2=0x7f8b880c25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:57.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.267+0000 7f8b8f382700 1 --2- 172.21.15.78:0/2269702908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b880c4540 0x7f8b880c25d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c00c540 tx=0x7f8b7c00c900 comp rx=0 tx=0).stop 2023-10-14T11:58:57.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.267+0000 7f8b8f382700 1 -- 172.21.15.78:0/2269702908 shutdown_connections 2023-10-14T11:58:57.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.267+0000 7f8b8f382700 1 --2- 172.21.15.78:0/2269702908 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8b6c06af00 0x7f8b6c06d3c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:57.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.267+0000 7f8b8f382700 1 --2- 172.21.15.78:0/2269702908 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8b8813dbe0 0x7f8b881ed870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:57.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.267+0000 7f8b8f382700 1 --2- 172.21.15.78:0/2269702908 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8b8806ae10 0x7f8b88138d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:57.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.267+0000 7f8b8f382700 1 --2- 172.21.15.78:0/2269702908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b880c4540 0x7f8b880c25d0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:57.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.267+0000 7f8b8f382700 1 -- 172.21.15.78:0/2269702908 >> 172.21.15.78:0/2269702908 conn(0x7f8b880c3a50 msgr2=0x7f8b880c02e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:57.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.267+0000 7f8b8f382700 1 -- 172.21.15.78:0/2269702908 shutdown_connections 2023-10-14T11:58:57.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.267+0000 7f8b8f382700 1 -- 172.21.15.78:0/2269702908 wait complete. 2023-10-14T11:58:57.280 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2023-10-14T11:58:57.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.400+0000 7f37d4034700 1 Processor -- start 2023-10-14T11:58:57.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.400+0000 7f37d4034700 1 -- start start 2023-10-14T11:58:57.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.401+0000 7f37d4034700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f37cc13dbe0 0x7f37cc06a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:57.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.401+0000 7f37d4034700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37cc06ebe0 con 0x7f37cc0c4540 2023-10-14T11:58:57.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.401+0000 7f37d4034700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37cc1558c0 con 0x7f37cc06ae10 2023-10-14T11:58:57.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.401+0000 7f37d4034700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37cc06d330 con 0x7f37cc13dbe0 2023-10-14T11:58:57.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.401+0000 7f37d15cf700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f37cc0c4540 0x7f37cc0c02e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50960/0 (socket says 172.21.15.78:50960) 2023-10-14T11:58:57.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.401+0000 7f37d15cf700 1 -- 172.21.15.78:0/1755003715 learned_addr learned my addr 172.21.15.78:0/1755003715 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:57.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.402+0000 7f37d1dd0700 1 --2- 172.21.15.78:0/1755003715 >> v2:172.21.15.195:3300/0 conn(0x7f37cc13dbe0 0x7f37cc06a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:57.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.402+0000 7f37d0dce700 1 -- 172.21.15.78:0/1755003715 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1196152823 0 0) 0x7f37cc06ebe0 con 0x7f37cc0c4540 2023-10-14T11:58:57.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.402+0000 7f37d0dce700 1 -- 172.21.15.78:0/1755003715 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37b80035b0 con 0x7f37cc0c4540 2023-10-14T11:58:57.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.402+0000 7f37d0dce700 1 -- 172.21.15.78:0/1755003715 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1810063406 0 0) 0x7f37b80035b0 con 0x7f37cc0c4540 2023-10-14T11:58:57.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.402+0000 7f37d0dce700 1 -- 172.21.15.78:0/1755003715 >> v2:172.21.15.195:3300/0 conn(0x7f37cc13dbe0 msgr2=0x7f37cc06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:57.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.402+0000 7f37d0dce700 1 --2- 172.21.15.78:0/1755003715 >> v2:172.21.15.195:3300/0 conn(0x7f37cc13dbe0 0x7f37cc06a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:57.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.402+0000 7f37d0dce700 1 -- 172.21.15.78:0/1755003715 >> v1:172.21.15.195:6789/0 conn(0x7f37cc06ae10 legacy=0x7f37cc15f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:57.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.402+0000 7f37d0dce700 1 -- 172.21.15.78:0/1755003715 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37cc154b00 con 0x7f37cc0c4540 2023-10-14T11:58:57.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.402+0000 7f37d0dce700 1 -- 172.21.15.78:0/1755003715 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441799917 0 0) 0x7f37bc001670 con 0x7f37cc0c4540 2023-10-14T11:58:57.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.402+0000 7f37d0dce700 1 -- 172.21.15.78:0/1755003715 >> v1:172.21.15.78:6789/0 conn(0x7f37cc0c4540 legacy=0x7f37cc0c02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:57.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.402+0000 7f37d0dce700 1 --2- 172.21.15.78:0/1755003715 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f37b8003a00 0x7f37b8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:57.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.402+0000 7f37d0dce700 1 --2- 172.21.15.78:0/1755003715 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f37b8004850 0x7f37b8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:57.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.402+0000 7f37d0dce700 1 --2- 172.21.15.78:0/1755003715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37b8005210 0x7f37b800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:57.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.402+0000 7f37d0dce700 1 -- 172.21.15.78:0/1755003715 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc06d330 con 0x7f37b8005210 2023-10-14T11:58:57.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.402+0000 7f37d0dce700 1 -- 172.21.15.78:0/1755003715 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b800c0b0 con 0x7f37b8003a00 2023-10-14T11:58:57.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.402+0000 7f37d0dce700 1 -- 172.21.15.78:0/1755003715 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37b800c270 con 0x7f37b8004850 2023-10-14T11:58:57.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.402+0000 7f37d25d1700 1 --2- 172.21.15.78:0/1755003715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37b8005210 0x7f37b800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:57.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.402+0000 7f37d1dd0700 1 --2- 172.21.15.78:0/1755003715 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f37b8003a00 0x7f37b8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:57.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.402+0000 7f37d15cf700 1 --2- 172.21.15.78:0/1755003715 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f37b8004850 0x7f37b8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:57.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.402+0000 7f37d25d1700 1 -- 172.21.15.78:0/1755003715 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f37b8004850 msgr2=0x7f37b8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:57.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.402+0000 7f37d25d1700 1 --2- 172.21.15.78:0/1755003715 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f37b8004850 0x7f37b8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:57.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.402+0000 7f37d25d1700 1 -- 172.21.15.78:0/1755003715 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f37b8003a00 msgr2=0x7f37b8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:57.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.402+0000 7f37d25d1700 1 --2- 172.21.15.78:0/1755003715 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f37b8003a00 0x7f37b8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:57.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.402+0000 7f37d25d1700 1 -- 172.21.15.78:0/1755003715 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b80035b0 con 0x7f37b8005210 2023-10-14T11:58:57.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.402+0000 7f37d25d1700 1 -- 172.21.15.78:0/1755003715 reap_dead start 2023-10-14T11:58:57.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.402+0000 7f37d25d1700 1 --2- 172.21.15.78:0/1755003715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37b8005210 0x7f37b800ba10 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f37c400c190 tx=0x7f37b8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86a163a980a8e123 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:57.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.403+0000 7f37d0dce700 1 -- 172.21.15.78:0/1755003715 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c4001db0 con 0x7f37b8005210 2023-10-14T11:58:57.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.403+0000 7f37d0dce700 1 -- 172.21.15.78:0/1755003715 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37c400ccf0 con 0x7f37b8005210 2023-10-14T11:58:57.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.403+0000 7f37d0dce700 1 -- 172.21.15.78:0/1755003715 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c4003c40 con 0x7f37b8005210 2023-10-14T11:58:57.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.403+0000 7f37d4034700 1 -- 172.21.15.78:0/1755003715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37b8005210 msgr2=0x7f37b800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:57.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.403+0000 7f37d4034700 1 --2- 172.21.15.78:0/1755003715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37b8005210 0x7f37b800ba10 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f37c400c190 tx=0x7f37b8005880 comp rx=0 tx=0).stop 2023-10-14T11:58:57.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.403+0000 7f37d4034700 1 -- 172.21.15.78:0/1755003715 shutdown_connections 2023-10-14T11:58:57.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.403+0000 7f37d4034700 1 --2- 172.21.15.78:0/1755003715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37b8005210 0x7f37b800ba10 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:57.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.403+0000 7f37d4034700 1 -- 172.21.15.78:0/1755003715 >> 172.21.15.78:0/1755003715 conn(0x7f37cc0c3a50 msgr2=0x7f37cc0c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:57.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.403+0000 7f37d4034700 1 -- 172.21.15.78:0/1755003715 shutdown_connections 2023-10-14T11:58:57.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.403+0000 7f37d4034700 1 -- 172.21.15.78:0/1755003715 wait complete. 2023-10-14T11:58:57.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.404+0000 7f37d4034700 1 Processor -- start 2023-10-14T11:58:57.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.404+0000 7f37d4034700 1 -- start start 2023-10-14T11:58:57.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.405+0000 7f37d4034700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f37b8005210 0x7f37cc1ea070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:57.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.405+0000 7f37d4034700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f37cc0c6090 0x7f37cc156320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:57.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.405+0000 7f37d4034700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37cc156860 0x7f37cc138a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:57.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.405+0000 7f37d4034700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc1ea5b0 con 0x7f37cc156860 2023-10-14T11:58:57.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.405+0000 7f37d4034700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc0a89c0 con 0x7f37b8005210 2023-10-14T11:58:57.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.405+0000 7f37d4034700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc155e90 con 0x7f37cc0c6090 2023-10-14T11:58:57.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.405+0000 7f37d25d1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37cc156860 0x7f37cc138a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:57.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.405+0000 7f37d25d1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37cc156860 0x7f37cc138a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46098/0 (socket says 172.21.15.78:46098) 2023-10-14T11:58:57.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.405+0000 7f37d25d1700 1 -- 172.21.15.78:0/4230547183 learned_addr learned my addr 172.21.15.78:0/4230547183 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:57.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.405+0000 7f37d15cf700 1 --2- 172.21.15.78:0/4230547183 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f37cc0c6090 0x7f37cc156320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:57.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.405+0000 7f37d1dd0700 1 --2- 172.21.15.78:0/4230547183 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f37b8005210 0x7f37cc1ea070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:57.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.406+0000 7f37d25d1700 1 -- 172.21.15.78:0/4230547183 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f37cc0c6090 msgr2=0x7f37cc156320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:57.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.406+0000 7f37d25d1700 1 --2- 172.21.15.78:0/4230547183 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f37cc0c6090 0x7f37cc156320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:57.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.406+0000 7f37d25d1700 1 -- 172.21.15.78:0/4230547183 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f37b8005210 msgr2=0x7f37cc1ea070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:57.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.406+0000 7f37d25d1700 1 --2- 172.21.15.78:0/4230547183 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f37b8005210 0x7f37cc1ea070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:57.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.406+0000 7f37d25d1700 1 -- 172.21.15.78:0/4230547183 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37cc156ce0 con 0x7f37cc156860 2023-10-14T11:58:57.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.406+0000 7f37d1dd0700 1 --2- 172.21.15.78:0/4230547183 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f37b8005210 0x7f37cc1ea070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:57.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.406+0000 7f37d25d1700 1 --2- 172.21.15.78:0/4230547183 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37cc156860 0x7f37cc138a40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f37cc152620 tx=0x7f37c40014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:57.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.406+0000 7f37c27fc700 1 -- 172.21.15.78:0/4230547183 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c4003e10 con 0x7f37cc156860 2023-10-14T11:58:57.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.406+0000 7f37c27fc700 1 -- 172.21.15.78:0/4230547183 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37c4005070 con 0x7f37cc156860 2023-10-14T11:58:57.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.406+0000 7f37d4034700 1 -- 172.21.15.78:0/4230547183 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37cc0a8c10 con 0x7f37cc156860 2023-10-14T11:58:57.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.406+0000 7f37c27fc700 1 -- 172.21.15.78:0/4230547183 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c40053e0 con 0x7f37cc156860 2023-10-14T11:58:57.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.406+0000 7f37d4034700 1 -- 172.21.15.78:0/4230547183 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37cc0a91d0 con 0x7f37cc156860 2023-10-14T11:58:57.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.407+0000 7f37d4034700 1 -- 172.21.15.78:0/4230547183 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37cc074b80 con 0x7f37cc156860 2023-10-14T11:58:57.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.407+0000 7f37c27fc700 1 -- 172.21.15.78:0/4230547183 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f37c4005540 con 0x7f37cc156860 2023-10-14T11:58:57.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.408+0000 7f37c27fc700 1 --2- 172.21.15.78:0/4230547183 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f37b006af00 0x7f37b006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:57.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.408+0000 7f37c27fc700 1 -- 172.21.15.78:0/4230547183 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f37c408a2f0 con 0x7f37cc156860 2023-10-14T11:58:57.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.408+0000 7f37d1dd0700 1 --2- 172.21.15.78:0/4230547183 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f37b006af00 0x7f37b006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:57.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.408+0000 7f37d1dd0700 1 --2- 172.21.15.78:0/4230547183 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f37b006af00 0x7f37b006d3c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f37c8009be0 tx=0x7f37c8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:57.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.412+0000 7f37c27fc700 1 -- 172.21.15.78:0/4230547183 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f37c4055780 con 0x7f37cc156860 2023-10-14T11:58:57.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.680+0000 7f37d4034700 1 -- 172.21.15.78:0/4230547183 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f37cc155fd0 con 0x7f37cc156860 2023-10-14T11:58:57.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.682+0000 7f37c27fc700 1 -- 172.21.15.78:0/4230547183 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v13) v1 ==== 97+0+0 (secure 0 0 0) 0x7f37c4058da0 con 0x7f37cc156860 2023-10-14T11:58:57.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.685+0000 7f37d4034700 1 -- 172.21.15.78:0/4230547183 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f37b006af00 msgr2=0x7f37b006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:57.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.685+0000 7f37d4034700 1 --2- 172.21.15.78:0/4230547183 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f37b006af00 0x7f37b006d3c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f37c8009be0 tx=0x7f37c8008040 comp rx=0 tx=0).stop 2023-10-14T11:58:57.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.685+0000 7f37d4034700 1 -- 172.21.15.78:0/4230547183 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37cc156860 msgr2=0x7f37cc138a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:57.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.685+0000 7f37d4034700 1 --2- 172.21.15.78:0/4230547183 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37cc156860 0x7f37cc138a40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f37cc152620 tx=0x7f37c40014a0 comp rx=0 tx=0).stop 2023-10-14T11:58:57.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.685+0000 7f37d4034700 1 -- 172.21.15.78:0/4230547183 shutdown_connections 2023-10-14T11:58:57.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.685+0000 7f37d4034700 1 --2- 172.21.15.78:0/4230547183 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37cc156860 0x7f37cc138a40 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:57.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.685+0000 7f37d4034700 1 --2- 172.21.15.78:0/4230547183 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f37b006af00 0x7f37b006d3c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:57.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.685+0000 7f37d4034700 1 --2- 172.21.15.78:0/4230547183 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f37b8005210 0x7f37cc1ea070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:57.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.685+0000 7f37d4034700 1 --2- 172.21.15.78:0/4230547183 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f37cc0c6090 0x7f37cc156320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:57.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.685+0000 7f37d4034700 1 -- 172.21.15.78:0/4230547183 >> 172.21.15.78:0/4230547183 conn(0x7f37cc0c3a50 msgr2=0x7f37cc13cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:57.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.685+0000 7f37d4034700 1 -- 172.21.15.78:0/4230547183 shutdown_connections 2023-10-14T11:58:57.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.685+0000 7f37d4034700 1 -- 172.21.15.78:0/4230547183 wait complete. 2023-10-14T11:58:57.698 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2023-10-14T11:58:57.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.819+0000 7f9e2ce8e700 1 Processor -- start 2023-10-14T11:58:57.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.819+0000 7f9e2ce8e700 1 -- start start 2023-10-14T11:58:57.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.820+0000 7f9e2ce8e700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9e2813dbe0 0x7f9e2806a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:57.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.820+0000 7f9e2ce8e700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9e280c4540 0x7f9e280c02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:57.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.820+0000 7f9e2ce8e700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9e2806ae10 0x7f9e280c0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:57.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.820+0000 7f9e2ce8e700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e2806d330 con 0x7f9e2806ae10 2023-10-14T11:58:57.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.820+0000 7f9e2ce8e700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e2806ebe0 con 0x7f9e2813dbe0 2023-10-14T11:58:57.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.820+0000 7f9e2ce8e700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e280c0d60 con 0x7f9e280c4540 2023-10-14T11:58:57.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.820+0000 7f9e26d9d700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9e2806ae10 0x7f9e280c0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:57.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.820+0000 7f9e26d9d700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9e2806ae10 0x7f9e280c0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46104/0 (socket says 172.21.15.78:46104) 2023-10-14T11:58:57.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.820+0000 7f9e26d9d700 1 -- 172.21.15.78:0/2006838946 learned_addr learned my addr 172.21.15.78:0/2006838946 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:57.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.820+0000 7f9e2659c700 1 --2- 172.21.15.78:0/2006838946 >> v2:172.21.15.195:3300/0 conn(0x7f9e2813dbe0 0x7f9e2806a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:57.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.820+0000 7f9e25d9b700 1 --2- 172.21.15.78:0/2006838946 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9e280c4540 0x7f9e280c02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:57.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.820+0000 7f9e26d9d700 1 -- 172.21.15.78:0/2006838946 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9e280c4540 msgr2=0x7f9e280c02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:57.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.820+0000 7f9e26d9d700 1 --2- 172.21.15.78:0/2006838946 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9e280c4540 0x7f9e280c02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:57.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.820+0000 7f9e26d9d700 1 -- 172.21.15.78:0/2006838946 >> v2:172.21.15.195:3300/0 conn(0x7f9e2813dbe0 msgr2=0x7f9e2806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:57.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.820+0000 7f9e26d9d700 1 --2- 172.21.15.78:0/2006838946 >> v2:172.21.15.195:3300/0 conn(0x7f9e2813dbe0 0x7f9e2806a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:57.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.820+0000 7f9e26d9d700 1 -- 172.21.15.78:0/2006838946 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e281558c0 con 0x7f9e2806ae10 2023-10-14T11:58:57.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.820+0000 7f9e25d9b700 1 --2- 172.21.15.78:0/2006838946 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9e280c4540 0x7f9e280c02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:57.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.820+0000 7f9e2659c700 1 --2- 172.21.15.78:0/2006838946 >> v2:172.21.15.195:3300/0 conn(0x7f9e2813dbe0 0x7f9e2806a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:57.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.820+0000 7f9e26d9d700 1 --2- 172.21.15.78:0/2006838946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e2806ae10 0x7f9e280c0820 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9e28072b60 tx=0x7f9e2000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c9349c88156429a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:57.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.820+0000 7f9e2559a700 1 -- 172.21.15.78:0/2006838946 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e2000e070 con 0x7f9e2806ae10 2023-10-14T11:58:57.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.820+0000 7f9e2559a700 1 -- 172.21.15.78:0/2006838946 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e2000b9c0 con 0x7f9e2806ae10 2023-10-14T11:58:57.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.820+0000 7f9e2559a700 1 -- 172.21.15.78:0/2006838946 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e2000bd70 con 0x7f9e2806ae10 2023-10-14T11:58:57.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.821+0000 7f9e2ce8e700 1 -- 172.21.15.78:0/2006838946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e2806ae10 msgr2=0x7f9e280c0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:57.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.821+0000 7f9e2ce8e700 1 --2- 172.21.15.78:0/2006838946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e2806ae10 0x7f9e280c0820 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9e28072b60 tx=0x7f9e2000b330 comp rx=0 tx=0).stop 2023-10-14T11:58:57.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.821+0000 7f9e2ce8e700 1 -- 172.21.15.78:0/2006838946 shutdown_connections 2023-10-14T11:58:57.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.821+0000 7f9e2ce8e700 1 --2- 172.21.15.78:0/2006838946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e2806ae10 0x7f9e280c0820 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:57.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.821+0000 7f9e2ce8e700 1 --2- 172.21.15.78:0/2006838946 >> v2:172.21.15.195:3300/0 conn(0x7f9e2813dbe0 0x7f9e2806a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:57.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.821+0000 7f9e2ce8e700 1 --2- 172.21.15.78:0/2006838946 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9e280c4540 0x7f9e280c02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:57.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.821+0000 7f9e2ce8e700 1 -- 172.21.15.78:0/2006838946 >> 172.21.15.78:0/2006838946 conn(0x7f9e280c3a50 msgr2=0x7f9e280c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:57.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.821+0000 7f9e2ce8e700 1 -- 172.21.15.78:0/2006838946 shutdown_connections 2023-10-14T11:58:57.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.821+0000 7f9e2ce8e700 1 -- 172.21.15.78:0/2006838946 wait complete. 2023-10-14T11:58:57.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.822+0000 7f9e2ce8e700 1 Processor -- start 2023-10-14T11:58:57.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.822+0000 7f9e2ce8e700 1 -- start start 2023-10-14T11:58:57.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.823+0000 7f9e2ce8e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e2806ae10 0x7f9e281588f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:57.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.823+0000 7f9e2ce8e700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9e280c4540 0x7f9e280c65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:57.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.823+0000 7f9e2ce8e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e2813dbe0 0x7f9e28155c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:57.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.823+0000 7f9e2ce8e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e28157bb0 con 0x7f9e2806ae10 2023-10-14T11:58:57.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.823+0000 7f9e2ce8e700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e281edc90 con 0x7f9e2813dbe0 2023-10-14T11:58:57.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.823+0000 7f9e2ce8e700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2806b760 con 0x7f9e280c4540 2023-10-14T11:58:57.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.823+0000 7f9e2659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e2806ae10 0x7f9e281588f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:57.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.823+0000 7f9e2659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e2806ae10 0x7f9e281588f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46106/0 (socket says 172.21.15.78:46106) 2023-10-14T11:58:57.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.823+0000 7f9e2659c700 1 -- 172.21.15.78:0/3929503518 learned_addr learned my addr 172.21.15.78:0/3929503518 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:57.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.823+0000 7f9e25d9b700 1 --2- 172.21.15.78:0/3929503518 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9e280c4540 0x7f9e280c65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:57.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.823+0000 7f9e26d9d700 1 --2- 172.21.15.78:0/3929503518 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e2813dbe0 0x7f9e28155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:57.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.823+0000 7f9e2659c700 1 -- 172.21.15.78:0/3929503518 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9e280c4540 msgr2=0x7f9e280c65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:57.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.823+0000 7f9e2659c700 1 --2- 172.21.15.78:0/3929503518 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9e280c4540 0x7f9e280c65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:57.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.823+0000 7f9e2659c700 1 -- 172.21.15.78:0/3929503518 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e2813dbe0 msgr2=0x7f9e28155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:57.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.823+0000 7f9e2659c700 1 --2- 172.21.15.78:0/3929503518 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e2813dbe0 0x7f9e28155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:57.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.823+0000 7f9e2659c700 1 -- 172.21.15.78:0/3929503518 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e20009b30 con 0x7f9e2806ae10 2023-10-14T11:58:57.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.823+0000 7f9e26d9d700 1 --2- 172.21.15.78:0/3929503518 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e2813dbe0 0x7f9e28155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:57.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.823+0000 7f9e2659c700 1 --2- 172.21.15.78:0/3929503518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e2806ae10 0x7f9e281588f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9e1800c780 tx=0x7f9e1800cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:57.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.824+0000 7f9e16ffd700 1 -- 172.21.15.78:0/3929503518 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e18011800 con 0x7f9e2806ae10 2023-10-14T11:58:57.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.824+0000 7f9e16ffd700 1 -- 172.21.15.78:0/3929503518 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e18011960 con 0x7f9e2806ae10 2023-10-14T11:58:57.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.824+0000 7f9e2ce8e700 1 -- 172.21.15.78:0/3929503518 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e280a8b60 con 0x7f9e2806ae10 2023-10-14T11:58:57.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.824+0000 7f9e16ffd700 1 -- 172.21.15.78:0/3929503518 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e18011cd0 con 0x7f9e2806ae10 2023-10-14T11:58:57.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.824+0000 7f9e2ce8e700 1 -- 172.21.15.78:0/3929503518 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e280a8e40 con 0x7f9e2806ae10 2023-10-14T11:58:57.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.824+0000 7f9e2ce8e700 1 -- 172.21.15.78:0/3929503518 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e28158740 con 0x7f9e2806ae10 2023-10-14T11:58:57.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.825+0000 7f9e16ffd700 1 -- 172.21.15.78:0/3929503518 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f9e18021350 con 0x7f9e2806ae10 2023-10-14T11:58:57.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.825+0000 7f9e16ffd700 1 --2- 172.21.15.78:0/3929503518 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f9e0806af00 0x7f9e0806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:57.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.826+0000 7f9e16ffd700 1 -- 172.21.15.78:0/3929503518 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9e18089150 con 0x7f9e2806ae10 2023-10-14T11:58:57.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.826+0000 7f9e25d9b700 1 --2- 172.21.15.78:0/3929503518 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f9e0806af00 0x7f9e0806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:57.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.826+0000 7f9e25d9b700 1 --2- 172.21.15.78:0/3929503518 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f9e0806af00 0x7f9e0806d3c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c006fd0 tx=0x7f9e1c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:57.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:57.830+0000 7f9e16ffd700 1 -- 172.21.15.78:0/3929503518 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9e18054560 con 0x7f9e2806ae10 2023-10-14T11:58:58.096 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.095+0000 7f9e2ce8e700 1 -- 172.21.15.78:0/3929503518 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f9e28074b80 con 0x7f9e2806ae10 2023-10-14T11:58:58.098 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.097+0000 7f9e16ffd700 1 -- 172.21.15.78:0/3929503518 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v14) v1 ==== 90+0+0 (secure 0 0 0) 0x7f9e18057b80 con 0x7f9e2806ae10 2023-10-14T11:58:58.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.099+0000 7f9e2ce8e700 1 -- 172.21.15.78:0/3929503518 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f9e0806af00 msgr2=0x7f9e0806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:58.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.099+0000 7f9e2ce8e700 1 --2- 172.21.15.78:0/3929503518 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f9e0806af00 0x7f9e0806d3c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c006fd0 tx=0x7f9e1c008040 comp rx=0 tx=0).stop 2023-10-14T11:58:58.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.099+0000 7f9e2ce8e700 1 -- 172.21.15.78:0/3929503518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e2806ae10 msgr2=0x7f9e281588f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:58.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.099+0000 7f9e2ce8e700 1 --2- 172.21.15.78:0/3929503518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e2806ae10 0x7f9e281588f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9e1800c780 tx=0x7f9e1800cb40 comp rx=0 tx=0).stop 2023-10-14T11:58:58.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.100+0000 7f9e2ce8e700 1 -- 172.21.15.78:0/3929503518 shutdown_connections 2023-10-14T11:58:58.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.100+0000 7f9e2ce8e700 1 --2- 172.21.15.78:0/3929503518 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f9e0806af00 0x7f9e0806d3c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:58.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.100+0000 7f9e2ce8e700 1 --2- 172.21.15.78:0/3929503518 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e2813dbe0 0x7f9e28155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:58.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.100+0000 7f9e2ce8e700 1 --2- 172.21.15.78:0/3929503518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e2806ae10 0x7f9e281588f0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:58.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.100+0000 7f9e2ce8e700 1 --2- 172.21.15.78:0/3929503518 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9e280c4540 0x7f9e280c65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:58.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.100+0000 7f9e2ce8e700 1 -- 172.21.15.78:0/3929503518 >> 172.21.15.78:0/3929503518 conn(0x7f9e280c3a50 msgr2=0x7f9e280c02e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:58.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.100+0000 7f9e2ce8e700 1 -- 172.21.15.78:0/3929503518 shutdown_connections 2023-10-14T11:58:58.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.100+0000 7f9e2ce8e700 1 -- 172.21.15.78:0/3929503518 wait complete. 2023-10-14T11:58:58.113 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T11:58:58.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.233+0000 7fcc14ae8700 1 Processor -- start 2023-10-14T11:58:58.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.233+0000 7fcc14ae8700 1 -- start start 2023-10-14T11:58:58.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.234+0000 7fcc14ae8700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcc100c72a0 0x7fcc100c7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:58.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.234+0000 7fcc14ae8700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc10156b20 con 0x7fcc100c44c0 2023-10-14T11:58:58.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.234+0000 7fcc14ae8700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc10155f10 con 0x7fcc100c2c90 2023-10-14T11:58:58.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.234+0000 7fcc14ae8700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc101561f0 con 0x7fcc100c72a0 2023-10-14T11:58:58.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.234+0000 7fcc0dd9b700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcc100c72a0 0x7fcc100c7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:58.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.234+0000 7fcc0e59c700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fcc100c44c0 0x7fcc100c86e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50990/0 (socket says 172.21.15.78:50990) 2023-10-14T11:58:58.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.234+0000 7fcc0e59c700 1 -- 172.21.15.78:0/3083037456 learned_addr learned my addr 172.21.15.78:0/3083037456 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:58.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.234+0000 7fcc0dd9b700 1 -- 172.21.15.78:0/3083037456 >> v1:172.21.15.195:6789/0 conn(0x7fcc100c2c90 legacy=0x7fcc100c30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:58.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.234+0000 7fcc0dd9b700 1 -- 172.21.15.78:0/3083037456 >> v1:172.21.15.78:6789/0 conn(0x7fcc100c44c0 legacy=0x7fcc100c86e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:58.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.234+0000 7fcc0dd9b700 1 -- 172.21.15.78:0/3083037456 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc101564d0 con 0x7fcc100c72a0 2023-10-14T11:58:58.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.234+0000 7fcc0d59a700 1 -- 172.21.15.78:0/3083037456 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3809125602 0 0) 0x7fcc10156b20 con 0x7fcc100c44c0 2023-10-14T11:58:58.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.234+0000 7fcc0dd9b700 1 --2- 172.21.15.78:0/3083037456 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc100c72a0 0x7fcc100c7a00 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fcc04009840 tx=0x7fcc04009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e412b5588038712a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:58.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.234+0000 7fcc0d59a700 1 -- 172.21.15.78:0/3083037456 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc04004230 con 0x7fcc100c72a0 2023-10-14T11:58:58.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.234+0000 7fcc0d59a700 1 -- 172.21.15.78:0/3083037456 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc04004390 con 0x7fcc100c72a0 2023-10-14T11:58:58.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.234+0000 7fcc0d59a700 1 -- 172.21.15.78:0/3083037456 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc040036a0 con 0x7fcc100c72a0 2023-10-14T11:58:58.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.235+0000 7fcc14ae8700 1 -- 172.21.15.78:0/3083037456 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc100c72a0 msgr2=0x7fcc100c7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:58.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.235+0000 7fcc14ae8700 1 --2- 172.21.15.78:0/3083037456 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc100c72a0 0x7fcc100c7a00 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fcc04009840 tx=0x7fcc04009b50 comp rx=0 tx=0).stop 2023-10-14T11:58:58.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.235+0000 7fcc14ae8700 1 -- 172.21.15.78:0/3083037456 shutdown_connections 2023-10-14T11:58:58.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.235+0000 7fcc14ae8700 1 --2- 172.21.15.78:0/3083037456 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc100c72a0 0x7fcc100c7a00 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:58.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.235+0000 7fcc14ae8700 1 -- 172.21.15.78:0/3083037456 >> 172.21.15.78:0/3083037456 conn(0x7fcc100737d0 msgr2=0x7fcc10073be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:58.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.235+0000 7fcc14ae8700 1 -- 172.21.15.78:0/3083037456 shutdown_connections 2023-10-14T11:58:58.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.235+0000 7fcc14ae8700 1 -- 172.21.15.78:0/3083037456 wait complete. 2023-10-14T11:58:58.247 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.236+0000 7fcc14ae8700 1 Processor -- start 2023-10-14T11:58:58.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.236+0000 7fcc14ae8700 1 -- start start 2023-10-14T11:58:58.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.237+0000 7fcc14ae8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc100c2c90 0x7fcc1006bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:58.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.237+0000 7fcc14ae8700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcc100c44c0 0x7fcc1006c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:58.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.237+0000 7fcc14ae8700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcc100c72a0 0x7fcc100ab790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:58.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.237+0000 7fcc14ae8700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc100aafd0 con 0x7fcc100c2c90 2023-10-14T11:58:58.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.237+0000 7fcc14ae8700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc101eae70 con 0x7fcc100c44c0 2023-10-14T11:58:58.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.237+0000 7fcc14ae8700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc100ab270 con 0x7fcc100c72a0 2023-10-14T11:58:58.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.237+0000 7fcc0e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc100c2c90 0x7fcc1006bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:58.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.237+0000 7fcc0e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc100c2c90 0x7fcc1006bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46120/0 (socket says 172.21.15.78:46120) 2023-10-14T11:58:58.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.237+0000 7fcc0e59c700 1 -- 172.21.15.78:0/2080003344 learned_addr learned my addr 172.21.15.78:0/2080003344 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:58.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.237+0000 7fcc0dd9b700 1 --2- 172.21.15.78:0/2080003344 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcc100c44c0 0x7fcc1006c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:58.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.237+0000 7fcc0ed9d700 1 --2- 172.21.15.78:0/2080003344 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcc100c72a0 0x7fcc100ab790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:58.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.237+0000 7fcc0e59c700 1 -- 172.21.15.78:0/2080003344 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcc100c72a0 msgr2=0x7fcc100ab790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:58.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.237+0000 7fcc0e59c700 1 --2- 172.21.15.78:0/2080003344 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcc100c72a0 0x7fcc100ab790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:58.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.237+0000 7fcc0e59c700 1 -- 172.21.15.78:0/2080003344 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcc100c44c0 msgr2=0x7fcc1006c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:58.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.237+0000 7fcc0e59c700 1 --2- 172.21.15.78:0/2080003344 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcc100c44c0 0x7fcc1006c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:58.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.237+0000 7fcc0e59c700 1 -- 172.21.15.78:0/2080003344 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc040094a0 con 0x7fcc100c2c90 2023-10-14T11:58:58.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.237+0000 7fcc0ed9d700 1 --2- 172.21.15.78:0/2080003344 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcc100c72a0 0x7fcc100ab790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:58.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.237+0000 7fcc0e59c700 1 --2- 172.21.15.78:0/2080003344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc100c2c90 0x7fcc1006bdb0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fcc0000c520 tx=0x7fcc0000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:58.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.238+0000 7fcbfeffd700 1 -- 172.21.15.78:0/2080003344 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc00004c80 con 0x7fcc100c2c90 2023-10-14T11:58:58.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.238+0000 7fcbfeffd700 1 -- 172.21.15.78:0/2080003344 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc00004e00 con 0x7fcc100c2c90 2023-10-14T11:58:58.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.238+0000 7fcc14ae8700 1 -- 172.21.15.78:0/2080003344 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc1006cea0 con 0x7fcc100c2c90 2023-10-14T11:58:58.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.238+0000 7fcbfeffd700 1 -- 172.21.15.78:0/2080003344 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc00007310 con 0x7fcc100c2c90 2023-10-14T11:58:58.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.238+0000 7fcc14ae8700 1 -- 172.21.15.78:0/2080003344 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc1006d210 con 0x7fcc100c2c90 2023-10-14T11:58:58.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.239+0000 7fcc14ae8700 1 -- 172.21.15.78:0/2080003344 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc10074b80 con 0x7fcc100c2c90 2023-10-14T11:58:58.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.239+0000 7fcbfeffd700 1 -- 172.21.15.78:0/2080003344 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fcc00007470 con 0x7fcc100c2c90 2023-10-14T11:58:58.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.239+0000 7fcbfeffd700 1 --2- 172.21.15.78:0/2080003344 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fcbf006af00 0x7fcbf006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:58.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.240+0000 7fcc0dd9b700 1 --2- 172.21.15.78:0/2080003344 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fcbf006af00 0x7fcbf006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:58.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.240+0000 7fcbfeffd700 1 -- 172.21.15.78:0/2080003344 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcc00088fd0 con 0x7fcc100c2c90 2023-10-14T11:58:58.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.240+0000 7fcc0dd9b700 1 --2- 172.21.15.78:0/2080003344 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fcbf006af00 0x7fcbf006d3c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcc04000c00 tx=0x7fcc0401c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:58.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.244+0000 7fcbfeffd700 1 -- 172.21.15.78:0/2080003344 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcc000543d0 con 0x7fcc100c2c90 2023-10-14T11:58:58.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.436+0000 7fcc14ae8700 1 -- 172.21.15.78:0/2080003344 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc100c7080 con 0x7fcc100c2c90 2023-10-14T11:58:58.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.437+0000 7fcbfeffd700 1 -- 172.21.15.78:0/2080003344 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+12833 (secure 0 0 0) 0x7fcc000579f0 con 0x7fcc100c2c90 2023-10-14T11:58:58.438 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:58.438 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":28,"fsid":"4c5dc134-3092-406a-a1e1-13e6d9e926e8","created":"2023-10-14T11:57:57.363750+0000","modified":"2023-10-14T11:58:55.151187+0000","last_up_change":"2023-10-14T11:58:06.982472+0000","last_in_change":"2023-10-14T11:57:59.821373+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T11:58:09.001200+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"625a5fc0-9474-4eae-b80e-14c27a74ad19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6801","nonce":3884969780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6803","nonce":3884969780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6807","nonce":3884969780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6805","nonce":3884969780}]},"public_addr":"172.21.15.78:6801/3884969780","cluster_addr":"172.21.15.78:6803/3884969780","heartbeat_back_addr":"172.21.15.78:6807/3884969780","heartbeat_front_addr":"172.21.15.78:6805/3884969780","state":["exists","up"]},{"osd":1,"uuid":"49d63db1-2981-49ab-9c40-c50fc744b982","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6809","nonce":2176928406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6811","nonce":2176928406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6815","nonce":2176928406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6813","nonce":2176928406}]},"public_addr":"172.21.15.78:6809/2176928406","cluster_addr":"172.21.15.78:6811/2176928406","heartbeat_back_addr":"172.21.15.78:6815/2176928406","heartbeat_front_addr":"172.21.15.78:6813/2176928406","state":["exists","up"]},{"osd":2,"uuid":"1d7d0b29-cbd1-4717-9fe6-765a8d4998b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6825","nonce":3647044577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6827","nonce":3647044577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6831","nonce":3647044577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6829","nonce":3647044577}]},"public_addr":"172.21.15.78:6825/3647044577","cluster_addr":"172.21.15.78:6827/3647044577","heartbeat_back_addr":"172.21.15.78:6831/3647044577","heartbeat_front_addr":"172.21.15.78:6829/3647044577","state":["exists","up"]},{"osd":3,"uuid":"2f75ded1-e9e2-40e7-99ae-45e1e425e1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6817","nonce":3584112288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6819","nonce":3584112288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6823","nonce":3584112288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6821","nonce":3584112288}]},"public_addr":"172.21.15.78:6817/3584112288","cluster_addr":"172.21.15.78:6819/3584112288","heartbeat_back_addr":"172.21.15.78:6823/3584112288","heartbeat_front_addr":"172.21.15.78:6821/3584112288","state":["exists","up"]},{"osd":4,"uuid":"5130b432-8358-46cb-8a31-3064c976f6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6801","nonce":3732756301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6803","nonce":3732756301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6807","nonce":3732756301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6805","nonce":3732756301}]},"public_addr":"172.21.15.195:6801/3732756301","cluster_addr":"172.21.15.195:6803/3732756301","heartbeat_back_addr":"172.21.15.195:6807/3732756301","heartbeat_front_addr":"172.21.15.195:6805/3732756301","state":["exists","up"]},{"osd":5,"uuid":"60e234a9-a506-4e40-9c19-49a58f3cf28c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6825","nonce":2857706411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6827","nonce":2857706411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6831","nonce":2857706411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6829","nonce":2857706411}]},"public_addr":"172.21.15.195:6825/2857706411","cluster_addr":"172.21.15.195:6827/2857706411","heartbeat_back_addr":"172.21.15.195:6831/2857706411","heartbeat_front_addr":"172.21.15.195:6829/2857706411","state":["exists","up"]},{"osd":6,"uuid":"df7d2dd6-26b6-4eb1-96e5-94d1f3955b33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6810","nonce":72554374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6813","nonce":72554374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6820","nonce":72554374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6817","nonce":72554374}]},"public_addr":"172.21.15.195:6810/72554374","cluster_addr":"172.21.15.195:6813/72554374","heartbeat_back_addr":"172.21.15.195:6820/72554374","heartbeat_front_addr":"172.21.15.195:6817/72554374","state":["exists","up"]},{"osd":7,"uuid":"47a9d4de-f56d-4aa4-9900-c4a9cf9b2f97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6809","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6811","nonce":482472979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6816","nonce":482472979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6823","nonce":482472979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6821","nonce":482472979}]},"public_addr":"172.21.15.195:6811/482472979","cluster_addr":"172.21.15.195:6816/482472979","heartbeat_back_addr":"172.21.15.195:6823/482472979","heartbeat_front_addr":"172.21.15.195:6821/482472979","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.754003+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.005994+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.043603+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.024506+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.062585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.033767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.994825+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.038300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T11:58:58.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.439+0000 7fcc14ae8700 1 -- 172.21.15.78:0/2080003344 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fcbf006af00 msgr2=0x7fcbf006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:58.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.439+0000 7fcc14ae8700 1 --2- 172.21.15.78:0/2080003344 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fcbf006af00 0x7fcbf006d3c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcc04000c00 tx=0x7fcc0401c040 comp rx=0 tx=0).stop 2023-10-14T11:58:58.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.439+0000 7fcc14ae8700 1 -- 172.21.15.78:0/2080003344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc100c2c90 msgr2=0x7fcc1006bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:58.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.439+0000 7fcc14ae8700 1 --2- 172.21.15.78:0/2080003344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc100c2c90 0x7fcc1006bdb0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fcc0000c520 tx=0x7fcc0000c8e0 comp rx=0 tx=0).stop 2023-10-14T11:58:58.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.439+0000 7fcc14ae8700 1 -- 172.21.15.78:0/2080003344 shutdown_connections 2023-10-14T11:58:58.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.439+0000 7fcc14ae8700 1 --2- 172.21.15.78:0/2080003344 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcc100c72a0 0x7fcc100ab790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:58.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.439+0000 7fcc14ae8700 1 --2- 172.21.15.78:0/2080003344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc100c2c90 0x7fcc1006bdb0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:58.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.439+0000 7fcc14ae8700 1 --2- 172.21.15.78:0/2080003344 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fcbf006af00 0x7fcbf006d3c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:58.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.439+0000 7fcc14ae8700 1 --2- 172.21.15.78:0/2080003344 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcc100c44c0 0x7fcc1006c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:58.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.439+0000 7fcc14ae8700 1 -- 172.21.15.78:0/2080003344 >> 172.21.15.78:0/2080003344 conn(0x7fcc100737d0 msgr2=0x7fcc10153340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:58.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.439+0000 7fcc14ae8700 1 -- 172.21.15.78:0/2080003344 shutdown_connections 2023-10-14T11:58:58.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.439+0000 7fcc14ae8700 1 -- 172.21.15.78:0/2080003344 wait complete. 2023-10-14T11:58:58.452 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T11:58:09.001200+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T11:58:58.452 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T11:58:58.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.575+0000 7f7618330700 1 Processor -- start 2023-10-14T11:58:58.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.575+0000 7f7618330700 1 -- start start 2023-10-14T11:58:58.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.576+0000 7f7618330700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f761013dbe0 0x7f761006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:58.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.576+0000 7f7618330700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f76100c4540 0x7f76100c02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:58.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.576+0000 7f7618330700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f761006ebe0 con 0x7f761006ae10 2023-10-14T11:58:58.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.576+0000 7f7618330700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f761006d330 con 0x7f76100c4540 2023-10-14T11:58:58.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.576+0000 7f7618330700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76101558c0 con 0x7f761013dbe0 2023-10-14T11:58:58.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.576+0000 7f76158cb700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f76100c4540 0x7f76100c02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:58.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.576+0000 7f76158cb700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f76100c4540 0x7f76100c02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46126/0 (socket says 172.21.15.78:46126) 2023-10-14T11:58:58.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.576+0000 7f76158cb700 1 -- 172.21.15.78:0/2447240466 learned_addr learned my addr 172.21.15.78:0/2447240466 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:58.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.576+0000 7f76160cc700 1 --2- 172.21.15.78:0/2447240466 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f761013dbe0 0x7f761006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:58.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.576+0000 7f76158cb700 1 -- 172.21.15.78:0/2447240466 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f761013dbe0 msgr2=0x7f761006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:58.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.576+0000 7f76158cb700 1 --2- 172.21.15.78:0/2447240466 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f761013dbe0 0x7f761006a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:58.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.576+0000 7f76158cb700 1 -- 172.21.15.78:0/2447240466 >> v1:172.21.15.195:6789/0 conn(0x7f761006ae10 legacy=0x7f76100c0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:58.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.576+0000 7f76158cb700 1 -- 172.21.15.78:0/2447240466 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7610154b00 con 0x7f76100c4540 2023-10-14T11:58:58.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.577+0000 7f76158cb700 1 --2- 172.21.15.78:0/2447240466 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76100c4540 0x7f76100c02e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f760c0098f0 tx=0x7f760c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9b29c62d3ab4d69 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:58.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.577+0000 7f76150ca700 1 -- 172.21.15.78:0/2447240466 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f760c00b8a0 con 0x7f76100c4540 2023-10-14T11:58:58.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.577+0000 7f76150ca700 1 -- 172.21.15.78:0/2447240466 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f760c00ba00 con 0x7f76100c4540 2023-10-14T11:58:58.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.577+0000 7f76150ca700 1 -- 172.21.15.78:0/2447240466 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f760c00bd90 con 0x7f76100c4540 2023-10-14T11:58:58.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.577+0000 7f7618330700 1 -- 172.21.15.78:0/2447240466 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76100c4540 msgr2=0x7f76100c02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:58.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.577+0000 7f7618330700 1 --2- 172.21.15.78:0/2447240466 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76100c4540 0x7f76100c02e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f760c0098f0 tx=0x7f760c009c00 comp rx=0 tx=0).stop 2023-10-14T11:58:58.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.577+0000 7f7618330700 1 -- 172.21.15.78:0/2447240466 shutdown_connections 2023-10-14T11:58:58.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.577+0000 7f7618330700 1 --2- 172.21.15.78:0/2447240466 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f761013dbe0 0x7f761006a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:58.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.577+0000 7f7618330700 1 --2- 172.21.15.78:0/2447240466 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76100c4540 0x7f76100c02e0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:58.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.577+0000 7f7618330700 1 -- 172.21.15.78:0/2447240466 >> 172.21.15.78:0/2447240466 conn(0x7f76100c3a50 msgr2=0x7f76100c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:58.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.577+0000 7f7618330700 1 -- 172.21.15.78:0/2447240466 shutdown_connections 2023-10-14T11:58:58.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.577+0000 7f7618330700 1 -- 172.21.15.78:0/2447240466 wait complete. 2023-10-14T11:58:58.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.579+0000 7f7618330700 1 Processor -- start 2023-10-14T11:58:58.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.579+0000 7f7618330700 1 -- start start 2023-10-14T11:58:58.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.579+0000 7f7618330700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f761006ae10 0x7f7610137d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:58.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.579+0000 7f7618330700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f76100c4540 0x7f7610139740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:58.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.579+0000 7f7618330700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f761013dbe0 0x7f7610156140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:58.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.579+0000 7f7618330700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76101ea940 con 0x7f761013dbe0 2023-10-14T11:58:58.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.579+0000 7f7618330700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7610138250 con 0x7f76100c4540 2023-10-14T11:58:58.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.579+0000 7f7618330700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7610155cf0 con 0x7f761006ae10 2023-10-14T11:58:58.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.579+0000 7f76168cd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f761013dbe0 0x7f7610156140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:58.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.579+0000 7f76168cd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f761013dbe0 0x7f7610156140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46134/0 (socket says 172.21.15.78:46134) 2023-10-14T11:58:58.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.579+0000 7f76168cd700 1 -- 172.21.15.78:0/525274582 learned_addr learned my addr 172.21.15.78:0/525274582 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:58.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.580+0000 7f76160cc700 1 --2- 172.21.15.78:0/525274582 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f761006ae10 0x7f7610137d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:58.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.580+0000 7f76158cb700 1 --2- 172.21.15.78:0/525274582 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f76100c4540 0x7f7610139740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:58.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.580+0000 7f76168cd700 1 -- 172.21.15.78:0/525274582 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f761006ae10 msgr2=0x7f7610137d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:58.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.580+0000 7f76168cd700 1 --2- 172.21.15.78:0/525274582 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f761006ae10 0x7f7610137d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:58.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.580+0000 7f76168cd700 1 -- 172.21.15.78:0/525274582 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f76100c4540 msgr2=0x7f7610139740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:58.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.580+0000 7f76168cd700 1 --2- 172.21.15.78:0/525274582 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f76100c4540 0x7f7610139740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:58.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.580+0000 7f76168cd700 1 -- 172.21.15.78:0/525274582 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f760c009550 con 0x7f761013dbe0 2023-10-14T11:58:58.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.580+0000 7f76160cc700 1 --2- 172.21.15.78:0/525274582 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f761006ae10 0x7f7610137d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:58:58.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.580+0000 7f76168cd700 1 --2- 172.21.15.78:0/525274582 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f761013dbe0 0x7f7610156140 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f760800cb40 tx=0x7f760800cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:58.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.580+0000 7f7606ffd700 1 -- 172.21.15.78:0/525274582 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7608003c00 con 0x7f761013dbe0 2023-10-14T11:58:58.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.580+0000 7f7606ffd700 1 -- 172.21.15.78:0/525274582 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7608003d60 con 0x7f761013dbe0 2023-10-14T11:58:58.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.580+0000 7f7618330700 1 -- 172.21.15.78:0/525274582 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76100a8630 con 0x7f761013dbe0 2023-10-14T11:58:58.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.580+0000 7f7606ffd700 1 -- 172.21.15.78:0/525274582 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f760800ab90 con 0x7f761013dbe0 2023-10-14T11:58:58.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.580+0000 7f7618330700 1 -- 172.21.15.78:0/525274582 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76100a8bf0 con 0x7f761013dbe0 2023-10-14T11:58:58.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.581+0000 7f7618330700 1 -- 172.21.15.78:0/525274582 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7610155e60 con 0x7f761013dbe0 2023-10-14T11:58:58.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.581+0000 7f7606ffd700 1 -- 172.21.15.78:0/525274582 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f760800acf0 con 0x7f761013dbe0 2023-10-14T11:58:58.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.582+0000 7f7606ffd700 1 --2- 172.21.15.78:0/525274582 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f75f406af00 0x7f75f406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:58.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.582+0000 7f7606ffd700 1 -- 172.21.15.78:0/525274582 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f76080892f0 con 0x7f761013dbe0 2023-10-14T11:58:58.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.582+0000 7f76160cc700 1 --2- 172.21.15.78:0/525274582 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f75f406af00 0x7f75f406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:58.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.582+0000 7f76160cc700 1 --2- 172.21.15.78:0/525274582 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f75f406af00 0x7f75f406d3c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7600007a50 tx=0x7f7600008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:58.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.586+0000 7f7606ffd700 1 -- 172.21.15.78:0/525274582 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7610155e60 con 0x7f761013dbe0 2023-10-14T11:58:58.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.773+0000 7f7618330700 1 -- 172.21.15.78:0/525274582 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7610155e60 con 0x7f761013dbe0 2023-10-14T11:58:58.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.773+0000 7f7606ffd700 1 -- 172.21.15.78:0/525274582 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7610155e60 con 0x7f761013dbe0 2023-10-14T11:58:58.775 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-10-14T11:58:58.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.775+0000 7f7618330700 1 -- 172.21.15.78:0/525274582 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f75f406af00 msgr2=0x7f75f406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:58.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.775+0000 7f7618330700 1 --2- 172.21.15.78:0/525274582 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f75f406af00 0x7f75f406d3c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7600007a50 tx=0x7f7600008040 comp rx=0 tx=0).stop 2023-10-14T11:58:58.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.775+0000 7f7618330700 1 -- 172.21.15.78:0/525274582 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f761013dbe0 msgr2=0x7f7610156140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:58.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.775+0000 7f7618330700 1 --2- 172.21.15.78:0/525274582 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f761013dbe0 0x7f7610156140 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f760800cb40 tx=0x7f760800cf00 comp rx=0 tx=0).stop 2023-10-14T11:58:58.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.775+0000 7f7618330700 1 -- 172.21.15.78:0/525274582 shutdown_connections 2023-10-14T11:58:58.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.775+0000 7f7618330700 1 --2- 172.21.15.78:0/525274582 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f761013dbe0 0x7f7610156140 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:58.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.775+0000 7f7618330700 1 --2- 172.21.15.78:0/525274582 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f761006ae10 0x7f7610137d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:58.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.775+0000 7f7618330700 1 --2- 172.21.15.78:0/525274582 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f75f406af00 0x7f75f406d3c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:58.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.775+0000 7f7618330700 1 --2- 172.21.15.78:0/525274582 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f76100c4540 0x7f7610139740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:58.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.775+0000 7f7618330700 1 -- 172.21.15.78:0/525274582 >> 172.21.15.78:0/525274582 conn(0x7f76100c3a50 msgr2=0x7f76100c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:58.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.775+0000 7f7618330700 1 -- 172.21.15.78:0/525274582 shutdown_connections 2023-10-14T11:58:58.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.775+0000 7f7618330700 1 -- 172.21.15.78:0/525274582 wait complete. 2023-10-14T11:58:58.788 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-14T11:58:58.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.906+0000 7fd974623700 1 Processor -- start 2023-10-14T11:58:58.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.906+0000 7fd974623700 1 -- start start 2023-10-14T11:58:58.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.907+0000 7fd974623700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd96c06ab40 0x7fd96c06f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:58.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.907+0000 7fd974623700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd96c0c0350 0x7fd96c13d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:58.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.907+0000 7fd974623700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd96c06b630 con 0x7fd96c1554e0 2023-10-14T11:58:58.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.907+0000 7fd974623700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd96c0c49f0 con 0x7fd96c06ab40 2023-10-14T11:58:58.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.907+0000 7fd974623700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd96c0726a0 con 0x7fd96c0c0350 2023-10-14T11:58:58.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.907+0000 7fd9723bf700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd96c06ab40 0x7fd96c06f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:58.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.907+0000 7fd9723bf700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd96c06ab40 0x7fd96c06f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46140/0 (socket says 172.21.15.78:46140) 2023-10-14T11:58:58.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.907+0000 7fd971bbe700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd96c1554e0 0x7fd96c06af20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51016/0 (socket says 172.21.15.78:51016) 2023-10-14T11:58:58.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.907+0000 7fd9723bf700 1 -- 172.21.15.78:0/3719204754 learned_addr learned my addr 172.21.15.78:0/3719204754 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:58.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.908+0000 7fd972bc0700 1 --2- 172.21.15.78:0/3719204754 >> v2:172.21.15.195:3300/0 conn(0x7fd96c0c0350 0x7fd96c13d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:58.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.908+0000 7fd9723bf700 1 -- 172.21.15.78:0/3719204754 >> v2:172.21.15.195:3300/0 conn(0x7fd96c0c0350 msgr2=0x7fd96c13d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:58.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.908+0000 7fd9723bf700 1 --2- 172.21.15.78:0/3719204754 >> v2:172.21.15.195:3300/0 conn(0x7fd96c0c0350 0x7fd96c13d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:58.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.908+0000 7fd9723bf700 1 -- 172.21.15.78:0/3719204754 >> v1:172.21.15.78:6789/0 conn(0x7fd96c1554e0 legacy=0x7fd96c06af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:58.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.908+0000 7fd9713bd700 1 -- 172.21.15.78:0/3719204754 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1149326199 0 0) 0x7fd96c06b630 con 0x7fd96c1554e0 2023-10-14T11:58:58.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.908+0000 7fd9723bf700 1 -- 172.21.15.78:0/3719204754 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd96c072980 con 0x7fd96c06ab40 2023-10-14T11:58:58.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.908+0000 7fd9723bf700 1 --2- 172.21.15.78:0/3719204754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd96c06ab40 0x7fd96c06f1e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd96c0c1860 tx=0x7fd95c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48a23232fe93ecd8 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:58.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.908+0000 7fd9713bd700 1 -- 172.21.15.78:0/3719204754 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95c00b870 con 0x7fd96c06ab40 2023-10-14T11:58:58.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.908+0000 7fd9713bd700 1 -- 172.21.15.78:0/3719204754 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd95c00b9d0 con 0x7fd96c06ab40 2023-10-14T11:58:58.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.908+0000 7fd9713bd700 1 -- 172.21.15.78:0/3719204754 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95c00bd60 con 0x7fd96c06ab40 2023-10-14T11:58:58.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.909+0000 7fd974623700 1 -- 172.21.15.78:0/3719204754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd96c06ab40 msgr2=0x7fd96c06f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:58.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.909+0000 7fd974623700 1 --2- 172.21.15.78:0/3719204754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd96c06ab40 0x7fd96c06f1e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd96c0c1860 tx=0x7fd95c009bd0 comp rx=0 tx=0).stop 2023-10-14T11:58:58.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.909+0000 7fd974623700 1 -- 172.21.15.78:0/3719204754 shutdown_connections 2023-10-14T11:58:58.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.909+0000 7fd974623700 1 --2- 172.21.15.78:0/3719204754 >> v2:172.21.15.195:3300/0 conn(0x7fd96c0c0350 0x7fd96c13d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:58.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.909+0000 7fd974623700 1 --2- 172.21.15.78:0/3719204754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd96c06ab40 0x7fd96c06f1e0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:58.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.909+0000 7fd974623700 1 -- 172.21.15.78:0/3719204754 >> 172.21.15.78:0/3719204754 conn(0x7fd96c0c3ac0 msgr2=0x7fd96c0c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:58.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.909+0000 7fd974623700 1 -- 172.21.15.78:0/3719204754 shutdown_connections 2023-10-14T11:58:58.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.909+0000 7fd974623700 1 -- 172.21.15.78:0/3719204754 wait complete. 2023-10-14T11:58:58.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.910+0000 7fd974623700 1 Processor -- start 2023-10-14T11:58:58.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.910+0000 7fd974623700 1 -- start start 2023-10-14T11:58:58.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.911+0000 7fd974623700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd96c06ab40 0x7fd96c134590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:58.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.911+0000 7fd974623700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd96c0c0350 0x7fd96c1530e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:58.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.911+0000 7fd974623700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd96c1554e0 0x7fd96c1e9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:58.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.911+0000 7fd974623700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c1e97d0 con 0x7fd96c1554e0 2023-10-14T11:58:58.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.911+0000 7fd974623700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c1edbb0 con 0x7fd96c0c0350 2023-10-14T11:58:58.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.911+0000 7fd974623700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c150fb0 con 0x7fd96c06ab40 2023-10-14T11:58:58.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.911+0000 7fd972bc0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd96c1554e0 0x7fd96c1e9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:58.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.911+0000 7fd972bc0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd96c1554e0 0x7fd96c1e9cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46148/0 (socket says 172.21.15.78:46148) 2023-10-14T11:58:58.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.911+0000 7fd972bc0700 1 -- 172.21.15.78:0/2504464242 learned_addr learned my addr 172.21.15.78:0/2504464242 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:58.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.911+0000 7fd9723bf700 1 --2- 172.21.15.78:0/2504464242 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd96c06ab40 0x7fd96c134590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:58.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.911+0000 7fd971bbe700 1 --2- 172.21.15.78:0/2504464242 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd96c0c0350 0x7fd96c1530e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:58.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.911+0000 7fd972bc0700 1 -- 172.21.15.78:0/2504464242 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd96c06ab40 msgr2=0x7fd96c134590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:58.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.911+0000 7fd972bc0700 1 --2- 172.21.15.78:0/2504464242 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd96c06ab40 0x7fd96c134590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:58.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.911+0000 7fd972bc0700 1 -- 172.21.15.78:0/2504464242 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd96c0c0350 msgr2=0x7fd96c1530e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:58.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.911+0000 7fd972bc0700 1 --2- 172.21.15.78:0/2504464242 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd96c0c0350 0x7fd96c1530e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:58.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.911+0000 7fd972bc0700 1 -- 172.21.15.78:0/2504464242 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd95c009550 con 0x7fd96c1554e0 2023-10-14T11:58:58.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.911+0000 7fd972bc0700 1 --2- 172.21.15.78:0/2504464242 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd96c1554e0 0x7fd96c1e9cd0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd964007e40 tx=0x7fd96400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:58.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.911+0000 7fd962ffd700 1 -- 172.21.15.78:0/2504464242 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96400ed30 con 0x7fd96c1554e0 2023-10-14T11:58:58.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.911+0000 7fd962ffd700 1 -- 172.21.15.78:0/2504464242 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd964014420 con 0x7fd96c1554e0 2023-10-14T11:58:58.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.911+0000 7fd974623700 1 -- 172.21.15.78:0/2504464242 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd96c151290 con 0x7fd96c1554e0 2023-10-14T11:58:58.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.911+0000 7fd962ffd700 1 -- 172.21.15.78:0/2504464242 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9640147d0 con 0x7fd96c1554e0 2023-10-14T11:58:58.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.911+0000 7fd974623700 1 -- 172.21.15.78:0/2504464242 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd96c155d80 con 0x7fd96c1554e0 2023-10-14T11:58:58.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.912+0000 7fd974623700 1 -- 172.21.15.78:0/2504464242 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd96c074b80 con 0x7fd96c1554e0 2023-10-14T11:58:58.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.913+0000 7fd962ffd700 1 -- 172.21.15.78:0/2504464242 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd964014930 con 0x7fd96c1554e0 2023-10-14T11:58:58.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.913+0000 7fd962ffd700 1 --2- 172.21.15.78:0/2504464242 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd95006af00 0x7fd95006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:58.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.913+0000 7fd9723bf700 1 --2- 172.21.15.78:0/2504464242 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd95006af00 0x7fd95006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:58.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.913+0000 7fd962ffd700 1 -- 172.21.15.78:0/2504464242 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd9640898a0 con 0x7fd96c1554e0 2023-10-14T11:58:58.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.913+0000 7fd9723bf700 1 --2- 172.21.15.78:0/2504464242 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd95006af00 0x7fd95006d3c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd95c000c00 tx=0x7fd95c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:58.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:58.917+0000 7fd962ffd700 1 -- 172.21.15.78:0/2504464242 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd964054d30 con 0x7fd96c1554e0 2023-10-14T11:58:59.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.110+0000 7fd974623700 1 -- 172.21.15.78:0/2504464242 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd96c066ce0 con 0x7fd96c1554e0 2023-10-14T11:58:59.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.110+0000 7fd962ffd700 1 -- 172.21.15.78:0/2504464242 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v28) v1 ==== 84+0+65 (secure 0 0 0) 0x7fd964058350 con 0x7fd96c1554e0 2023-10-14T11:58:59.112 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:58:59.112 INFO:teuthology.orchestra.run.smithi078.stdout:[ 2023-10-14T11:58:59.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-14T11:58:59.113 INFO:teuthology.orchestra.run.smithi078.stdout: "poolnum": 1, 2023-10-14T11:58:59.113 INFO:teuthology.orchestra.run.smithi078.stdout: "poolname": ".mgr" 2023-10-14T11:58:59.114 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-14T11:58:59.114 INFO:teuthology.orchestra.run.smithi078.stdout:] 2023-10-14T11:58:59.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.112+0000 7fd974623700 1 -- 172.21.15.78:0/2504464242 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd95006af00 msgr2=0x7fd95006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:59.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.112+0000 7fd974623700 1 --2- 172.21.15.78:0/2504464242 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd95006af00 0x7fd95006d3c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd95c000c00 tx=0x7fd95c008030 comp rx=0 tx=0).stop 2023-10-14T11:58:59.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.112+0000 7fd974623700 1 -- 172.21.15.78:0/2504464242 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd96c1554e0 msgr2=0x7fd96c1e9cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:59.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.112+0000 7fd974623700 1 --2- 172.21.15.78:0/2504464242 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd96c1554e0 0x7fd96c1e9cd0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd964007e40 tx=0x7fd96400e3c0 comp rx=0 tx=0).stop 2023-10-14T11:58:59.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.112+0000 7fd974623700 1 -- 172.21.15.78:0/2504464242 shutdown_connections 2023-10-14T11:58:59.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.112+0000 7fd974623700 1 --2- 172.21.15.78:0/2504464242 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd96c1554e0 0x7fd96c1e9cd0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:59.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.112+0000 7fd974623700 1 --2- 172.21.15.78:0/2504464242 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd96c06ab40 0x7fd96c134590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:59.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.112+0000 7fd974623700 1 --2- 172.21.15.78:0/2504464242 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd95006af00 0x7fd95006d3c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:59.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.112+0000 7fd974623700 1 --2- 172.21.15.78:0/2504464242 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd96c0c0350 0x7fd96c1530e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:59.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.112+0000 7fd974623700 1 -- 172.21.15.78:0/2504464242 >> 172.21.15.78:0/2504464242 conn(0x7fd96c0c3ac0 msgr2=0x7fd96c0c2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:59.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.112+0000 7fd974623700 1 -- 172.21.15.78:0/2504464242 shutdown_connections 2023-10-14T11:58:59.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.112+0000 7fd974623700 1 -- 172.21.15.78:0/2504464242 wait complete. 2023-10-14T11:58:59.125 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'testcephfs1' 2023-10-14T11:58:59.126 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create testcephfs1_metadata --pg_num_min 64 2023-10-14T11:58:59.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.250+0000 7f8e9be31700 1 Processor -- start 2023-10-14T11:58:59.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.250+0000 7f8e9be31700 1 -- start start 2023-10-14T11:58:59.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.251+0000 7f8e9be31700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8e940c1340 0x7f8e9406df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:59.255 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.251+0000 7f8e9be31700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8e940a9d00 0x7f8e9406e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:59.255 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.251+0000 7f8e9be31700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e941521d0 con 0x7f8e940c1720 2023-10-14T11:58:59.256 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.251+0000 7f8e9be31700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e9406db50 con 0x7f8e940c1340 2023-10-14T11:58:59.256 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.251+0000 7f8e9be31700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e94152700 con 0x7f8e940a9d00 2023-10-14T11:58:59.256 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.251+0000 7f8e99bcd700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8e940c1340 0x7f8e9406df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:59.257 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.251+0000 7f8e9a3ce700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f8e940c1720 0x7f8e9414e8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51032/0 (socket says 172.21.15.78:51032) 2023-10-14T11:58:59.257 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.251+0000 7f8e9a3ce700 1 -- 172.21.15.78:0/927510815 learned_addr learned my addr 172.21.15.78:0/927510815 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:58:59.257 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.251+0000 7f8e993cc700 1 --2- 172.21.15.78:0/927510815 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8e940a9d00 0x7f8e9406e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:59.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.251+0000 7f8e99bcd700 1 -- 172.21.15.78:0/927510815 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8e940a9d00 msgr2=0x7f8e9406e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:58:59.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.251+0000 7f8e99bcd700 1 --2- 172.21.15.78:0/927510815 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8e940a9d00 0x7f8e9406e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:59.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.251+0000 7f8e99bcd700 1 -- 172.21.15.78:0/927510815 >> v1:172.21.15.78:6789/0 conn(0x7f8e940c1720 legacy=0x7f8e9414e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:59.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.251+0000 7f8e99bcd700 1 -- 172.21.15.78:0/927510815 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e940c1c80 con 0x7f8e940c1340 2023-10-14T11:58:59.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.251+0000 7f8e98bcb700 1 -- 172.21.15.78:0/927510815 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3750750951 0 0) 0x7f8e941521d0 con 0x7f8e940c1720 2023-10-14T11:58:59.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.251+0000 7f8e993cc700 1 --2- 172.21.15.78:0/927510815 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8e940a9d00 0x7f8e9406e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:58:59.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.251+0000 7f8e99bcd700 1 --2- 172.21.15.78:0/927510815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e940c1340 0x7f8e9406df70 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8e90009840 tx=0x7f8e90009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63b7e10a6b50d96b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:59.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.251+0000 7f8e98bcb700 1 -- 172.21.15.78:0/927510815 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e90011830 con 0x7f8e940c1340 2023-10-14T11:58:59.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.251+0000 7f8e98bcb700 1 -- 172.21.15.78:0/927510815 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e90011990 con 0x7f8e940c1340 2023-10-14T11:58:59.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.251+0000 7f8e98bcb700 1 -- 172.21.15.78:0/927510815 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e90011d20 con 0x7f8e940c1340 2023-10-14T11:58:59.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.252+0000 7f8e9be31700 1 -- 172.21.15.78:0/927510815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e940c1340 msgr2=0x7f8e9406df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:59.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.252+0000 7f8e9be31700 1 --2- 172.21.15.78:0/927510815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e940c1340 0x7f8e9406df70 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8e90009840 tx=0x7f8e90009b50 comp rx=0 tx=0).stop 2023-10-14T11:58:59.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.252+0000 7f8e9be31700 1 -- 172.21.15.78:0/927510815 shutdown_connections 2023-10-14T11:58:59.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.252+0000 7f8e9be31700 1 --2- 172.21.15.78:0/927510815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e940c1340 0x7f8e9406df70 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:59.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.252+0000 7f8e9be31700 1 --2- 172.21.15.78:0/927510815 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8e940a9d00 0x7f8e9406e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:59.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.252+0000 7f8e9be31700 1 -- 172.21.15.78:0/927510815 >> 172.21.15.78:0/927510815 conn(0x7f8e940742a0 msgr2=0x7f8e940a8490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:58:59.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.252+0000 7f8e9be31700 1 -- 172.21.15.78:0/927510815 shutdown_connections 2023-10-14T11:58:59.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.252+0000 7f8e9be31700 1 -- 172.21.15.78:0/927510815 wait complete. 2023-10-14T11:58:59.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.254+0000 7f8e9be31700 1 Processor -- start 2023-10-14T11:58:59.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.254+0000 7f8e9be31700 1 -- start start 2023-10-14T11:58:59.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.254+0000 7f8e9be31700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e940a9d00 0x7f8e94154920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:59.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.254+0000 7f8e9be31700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8e940c1340 0x7f8e940c5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:59.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.254+0000 7f8e9be31700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8e940c1720 0x7f8e94153120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:59.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.254+0000 7f8e9be31700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e94155b00 con 0x7f8e940a9d00 2023-10-14T11:58:59.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.255+0000 7f8e9be31700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e941e9410 con 0x7f8e940c1340 2023-10-14T11:58:59.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.255+0000 7f8e9be31700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e941e96f0 con 0x7f8e940c1720 2023-10-14T11:58:59.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.255+0000 7f8e99bcd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e940a9d00 0x7f8e94154920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:59.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.255+0000 7f8e99bcd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e940a9d00 0x7f8e94154920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46158/0 (socket says 172.21.15.78:46158) 2023-10-14T11:58:59.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.255+0000 7f8e99bcd700 1 -- 172.21.15.78:0/1971852992 learned_addr learned my addr 172.21.15.78:0/1971852992 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:58:59.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.255+0000 7f8e99bcd700 1 -- 172.21.15.78:0/1971852992 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8e940c1720 msgr2=0x7f8e94153120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:59.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.255+0000 7f8e99bcd700 1 --2- 172.21.15.78:0/1971852992 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8e940c1720 0x7f8e94153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:59.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.255+0000 7f8e99bcd700 1 -- 172.21.15.78:0/1971852992 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8e940c1340 msgr2=0x7f8e940c5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:58:59.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.255+0000 7f8e99bcd700 1 --2- 172.21.15.78:0/1971852992 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8e940c1340 0x7f8e940c5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:58:59.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.255+0000 7f8e99bcd700 1 -- 172.21.15.78:0/1971852992 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e900094a0 con 0x7f8e940a9d00 2023-10-14T11:58:59.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.255+0000 7f8e99bcd700 1 --2- 172.21.15.78:0/1971852992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e940a9d00 0x7f8e94154920 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8e900090b0 tx=0x7f8e90004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:59.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.255+0000 7f8e8a7fc700 1 -- 172.21.15.78:0/1971852992 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e900038a0 con 0x7f8e940a9d00 2023-10-14T11:58:59.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.255+0000 7f8e8a7fc700 1 -- 172.21.15.78:0/1971852992 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e9001f070 con 0x7f8e940a9d00 2023-10-14T11:58:59.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.255+0000 7f8e9be31700 1 -- 172.21.15.78:0/1971852992 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e941e99d0 con 0x7f8e940a9d00 2023-10-14T11:58:59.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.255+0000 7f8e8a7fc700 1 -- 172.21.15.78:0/1971852992 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e90003c40 con 0x7f8e940a9d00 2023-10-14T11:58:59.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.255+0000 7f8e9be31700 1 -- 172.21.15.78:0/1971852992 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e941e9f90 con 0x7f8e940a9d00 2023-10-14T11:58:59.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.256+0000 7f8e9be31700 1 -- 172.21.15.78:0/1971852992 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e94074b80 con 0x7f8e940a9d00 2023-10-14T11:58:59.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.256+0000 7f8e8a7fc700 1 -- 172.21.15.78:0/1971852992 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8e9001a430 con 0x7f8e940a9d00 2023-10-14T11:58:59.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.257+0000 7f8e8a7fc700 1 --2- 172.21.15.78:0/1971852992 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8e7806af00 0x7f8e7806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:58:59.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.257+0000 7f8e8a7fc700 1 -- 172.21.15.78:0/1971852992 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8e90021070 con 0x7f8e940a9d00 2023-10-14T11:58:59.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.257+0000 7f8e993cc700 1 --2- 172.21.15.78:0/1971852992 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8e7806af00 0x7f8e7806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:58:59.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.257+0000 7f8e993cc700 1 --2- 172.21.15.78:0/1971852992 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8e7806af00 0x7f8e7806d3c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8e94145790 tx=0x7f8e84008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:58:59.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.261+0000 7f8e8a7fc700 1 -- 172.21.15.78:0/1971852992 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8e900561d0 con 0x7f8e940a9d00 2023-10-14T11:58:59.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:58:59.452+0000 7f8e9be31700 1 -- 172.21.15.78:0/1971852992 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "testcephfs1_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f8e9406bf80 con 0x7f8e940a9d00 2023-10-14T11:59:00.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.155+0000 7f8e8a7fc700 1 -- 172.21.15.78:0/1971852992 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "testcephfs1_metadata", "pg_num_min": 64}]=0 pool 'testcephfs1_metadata' created v29) v1 ==== 148+0+0 (secure 0 0 0) 0x7f8e9002b0b0 con 0x7f8e940a9d00 2023-10-14T11:59:00.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.157+0000 7f8e9be31700 1 -- 172.21.15.78:0/1971852992 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8e7806af00 msgr2=0x7f8e7806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:00.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.157+0000 7f8e9be31700 1 --2- 172.21.15.78:0/1971852992 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8e7806af00 0x7f8e7806d3c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8e94145790 tx=0x7f8e84008040 comp rx=0 tx=0).stop 2023-10-14T11:59:00.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.157+0000 7f8e9be31700 1 -- 172.21.15.78:0/1971852992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e940a9d00 msgr2=0x7f8e94154920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:00.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.157+0000 7f8e9be31700 1 --2- 172.21.15.78:0/1971852992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e940a9d00 0x7f8e94154920 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8e900090b0 tx=0x7f8e90004750 comp rx=0 tx=0).stop 2023-10-14T11:59:00.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.157+0000 7f8e9be31700 1 -- 172.21.15.78:0/1971852992 shutdown_connections 2023-10-14T11:59:00.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.157+0000 7f8e9be31700 1 --2- 172.21.15.78:0/1971852992 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8e940c1720 0x7f8e94153120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:00.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.157+0000 7f8e9be31700 1 --2- 172.21.15.78:0/1971852992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e940a9d00 0x7f8e94154920 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:00.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.157+0000 7f8e9be31700 1 --2- 172.21.15.78:0/1971852992 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8e7806af00 0x7f8e7806d3c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:00.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.157+0000 7f8e9be31700 1 --2- 172.21.15.78:0/1971852992 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8e940c1340 0x7f8e940c5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:00.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.157+0000 7f8e9be31700 1 -- 172.21.15.78:0/1971852992 >> 172.21.15.78:0/1971852992 conn(0x7f8e940742a0 msgr2=0x7f8e9414ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:00.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.157+0000 7f8e9be31700 1 -- 172.21.15.78:0/1971852992 shutdown_connections 2023-10-14T11:59:00.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.157+0000 7f8e9be31700 1 -- 172.21.15.78:0/1971852992 wait complete. 2023-10-14T11:59:00.160 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'testcephfs1_metadata' created 2023-10-14T11:59:00.174 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create testcephfs1_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-14T11:59:00.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.301+0000 7f98458a5700 1 Processor -- start 2023-10-14T11:59:00.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.301+0000 7f98458a5700 1 -- start start 2023-10-14T11:59:00.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.302+0000 7f98458a5700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f984014ed10 0x7f9840152ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:00.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.302+0000 7f98458a5700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98400c26d0 con 0x7f98400c35e0 2023-10-14T11:59:00.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.302+0000 7f98458a5700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98400c29b0 con 0x7f9840154c20 2023-10-14T11:59:00.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.302+0000 7f98458a5700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98400a8520 con 0x7f984014ed10 2023-10-14T11:59:00.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.302+0000 7f983e7fc700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f98400c35e0 0x7f9840153400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51044/0 (socket says 172.21.15.78:51044) 2023-10-14T11:59:00.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.302+0000 7f983e7fc700 1 -- 172.21.15.78:0/3018351562 learned_addr learned my addr 172.21.15.78:0/3018351562 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:59:00.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.302+0000 7f983effd700 1 --2- 172.21.15.78:0/3018351562 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f984014ed10 0x7f9840152ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:00.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.302+0000 7f983dffb700 1 -- 172.21.15.78:0/3018351562 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 481873772 0 0) 0x7f98400c26d0 con 0x7f98400c35e0 2023-10-14T11:59:00.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.302+0000 7f983dffb700 1 -- 172.21.15.78:0/3018351562 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98280035b0 con 0x7f98400c35e0 2023-10-14T11:59:00.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.303+0000 7f983dffb700 1 -- 172.21.15.78:0/3018351562 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1373013805 0 0) 0x7f98280035b0 con 0x7f98400c35e0 2023-10-14T11:59:00.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.303+0000 7f983dffb700 1 -- 172.21.15.78:0/3018351562 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f984014ed10 msgr2=0x7f9840152ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:00.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.303+0000 7f983dffb700 1 --2- 172.21.15.78:0/3018351562 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f984014ed10 0x7f9840152ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:00.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.303+0000 7f983dffb700 1 -- 172.21.15.78:0/3018351562 >> v1:172.21.15.195:6789/0 conn(0x7f9840154c20 legacy=0x7f9840155090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:00.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.303+0000 7f983dffb700 1 -- 172.21.15.78:0/3018351562 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98400c2c90 con 0x7f98400c35e0 2023-10-14T11:59:00.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.303+0000 7f983dffb700 1 -- 172.21.15.78:0/3018351562 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441799917 0 0) 0x7f9830001670 con 0x7f98400c35e0 2023-10-14T11:59:00.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.303+0000 7f983dffb700 1 -- 172.21.15.78:0/3018351562 >> v1:172.21.15.78:6789/0 conn(0x7f98400c35e0 legacy=0x7f9840153400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:00.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.303+0000 7f983dffb700 1 --2- 172.21.15.78:0/3018351562 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9828003a30 0x7f9828003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:00.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.303+0000 7f983dffb700 1 --2- 172.21.15.78:0/3018351562 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f984014ed10 0x7f9828004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:00.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.303+0000 7f983dffb700 1 --2- 172.21.15.78:0/3018351562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9828004d90 0x7f9828005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:00.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.303+0000 7f983dffb700 1 -- 172.21.15.78:0/3018351562 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98400a8520 con 0x7f9828004d90 2023-10-14T11:59:00.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.303+0000 7f983dffb700 1 -- 172.21.15.78:0/3018351562 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982800bac0 con 0x7f9828003a30 2023-10-14T11:59:00.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.303+0000 7f983dffb700 1 -- 172.21.15.78:0/3018351562 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f982800bc80 con 0x7f984014ed10 2023-10-14T11:59:00.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.303+0000 7f983f7fe700 1 --2- 172.21.15.78:0/3018351562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9828004d90 0x7f9828005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:00.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.303+0000 7f983effd700 1 --2- 172.21.15.78:0/3018351562 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9828003a30 0x7f9828003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:00.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.303+0000 7f983e7fc700 1 --2- 172.21.15.78:0/3018351562 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f984014ed10 0x7f9828004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:00.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.303+0000 7f983f7fe700 1 -- 172.21.15.78:0/3018351562 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f984014ed10 msgr2=0x7f9828004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:00.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.303+0000 7f983f7fe700 1 --2- 172.21.15.78:0/3018351562 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f984014ed10 0x7f9828004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:00.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.303+0000 7f983f7fe700 1 -- 172.21.15.78:0/3018351562 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9828003a30 msgr2=0x7f9828003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:00.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.303+0000 7f983f7fe700 1 --2- 172.21.15.78:0/3018351562 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9828003a30 0x7f9828003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:00.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.303+0000 7f983f7fe700 1 -- 172.21.15.78:0/3018351562 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98280035b0 con 0x7f9828004d90 2023-10-14T11:59:00.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.303+0000 7f983f7fe700 1 --2- 172.21.15.78:0/3018351562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9828004d90 0x7f9828005210 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f983800c240 tx=0x7f9828005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ac2f29efe83e940 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:00.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.303+0000 7f983dffb700 1 -- 172.21.15.78:0/3018351562 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9838001e70 con 0x7f9828004d90 2023-10-14T11:59:00.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.303+0000 7f983dffb700 1 -- 172.21.15.78:0/3018351562 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f983800cde0 con 0x7f9828004d90 2023-10-14T11:59:00.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.303+0000 7f983dffb700 1 -- 172.21.15.78:0/3018351562 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9838012660 con 0x7f9828004d90 2023-10-14T11:59:00.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.304+0000 7f98458a5700 1 -- 172.21.15.78:0/3018351562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9828004d90 msgr2=0x7f9828005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:00.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.304+0000 7f98458a5700 1 --2- 172.21.15.78:0/3018351562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9828004d90 0x7f9828005210 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f983800c240 tx=0x7f9828005880 comp rx=0 tx=0).stop 2023-10-14T11:59:00.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.304+0000 7f983f7fe700 1 -- 172.21.15.78:0/3018351562 reap_dead start 2023-10-14T11:59:00.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.304+0000 7f98458a5700 1 -- 172.21.15.78:0/3018351562 shutdown_connections 2023-10-14T11:59:00.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.304+0000 7f98458a5700 1 -- 172.21.15.78:0/3018351562 >> 172.21.15.78:0/3018351562 conn(0x7f98400c3a50 msgr2=0x7f98400c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:00.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.304+0000 7f98458a5700 1 -- 172.21.15.78:0/3018351562 shutdown_connections 2023-10-14T11:59:00.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.304+0000 7f98458a5700 1 -- 172.21.15.78:0/3018351562 wait complete. 2023-10-14T11:59:00.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.305+0000 7f98458a5700 1 Processor -- start 2023-10-14T11:59:00.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.305+0000 7f98458a5700 1 -- start start 2023-10-14T11:59:00.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.306+0000 7f98458a5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f984014e0b0 0x7f9840151460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:00.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.306+0000 7f98458a5700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9840148660 0x7f9840148ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:00.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.306+0000 7f98458a5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f98401e9620 0x7f98401e9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:00.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.306+0000 7f98458a5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f984014f8f0 con 0x7f984014e0b0 2023-10-14T11:59:00.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.306+0000 7f98458a5700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98400a9bc0 con 0x7f98401e9620 2023-10-14T11:59:00.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.306+0000 7f98458a5700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9840149020 con 0x7f9840148660 2023-10-14T11:59:00.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.306+0000 7f983effd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f984014e0b0 0x7f9840151460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:00.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.306+0000 7f983effd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f984014e0b0 0x7f9840151460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46178/0 (socket says 172.21.15.78:46178) 2023-10-14T11:59:00.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.306+0000 7f983effd700 1 -- 172.21.15.78:0/362520961 learned_addr learned my addr 172.21.15.78:0/362520961 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:00.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.306+0000 7f983e7fc700 1 --2- 172.21.15.78:0/362520961 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9840148660 0x7f9840148ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:00.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.306+0000 7f983f7fe700 1 --2- 172.21.15.78:0/362520961 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f98401e9620 0x7f98401e9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:00.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.306+0000 7f983effd700 1 -- 172.21.15.78:0/362520961 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9840148660 msgr2=0x7f9840148ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:00.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.306+0000 7f983effd700 1 --2- 172.21.15.78:0/362520961 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9840148660 0x7f9840148ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:00.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.306+0000 7f983effd700 1 -- 172.21.15.78:0/362520961 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f98401e9620 msgr2=0x7f98401e9ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:00.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.306+0000 7f983effd700 1 --2- 172.21.15.78:0/362520961 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f98401e9620 0x7f98401e9ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:00.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.306+0000 7f983effd700 1 -- 172.21.15.78:0/362520961 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9840151d80 con 0x7f984014e0b0 2023-10-14T11:59:00.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.306+0000 7f983e7fc700 1 --2- 172.21.15.78:0/362520961 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9840148660 0x7f9840148ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:59:00.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.306+0000 7f983f7fe700 1 --2- 172.21.15.78:0/362520961 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f98401e9620 0x7f98401e9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:59:00.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.307+0000 7f983effd700 1 --2- 172.21.15.78:0/362520961 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f984014e0b0 0x7f9840151460 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f9834011b00 tx=0x7f9834011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:00.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.307+0000 7f9827fff700 1 -- 172.21.15.78:0/362520961 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983400fa00 con 0x7f984014e0b0 2023-10-14T11:59:00.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.307+0000 7f9827fff700 1 -- 172.21.15.78:0/362520961 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f983400fb60 con 0x7f984014e0b0 2023-10-14T11:59:00.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.307+0000 7f9827fff700 1 -- 172.21.15.78:0/362520961 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9834014440 con 0x7f984014e0b0 2023-10-14T11:59:00.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.307+0000 7f98458a5700 1 -- 172.21.15.78:0/362520961 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9840152060 con 0x7f984014e0b0 2023-10-14T11:59:00.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.307+0000 7f98458a5700 1 -- 172.21.15.78:0/362520961 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98400a9850 con 0x7f984014e0b0 2023-10-14T11:59:00.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.308+0000 7f98458a5700 1 -- 172.21.15.78:0/362520961 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9840074b80 con 0x7f984014e0b0 2023-10-14T11:59:00.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.308+0000 7f9827fff700 1 -- 172.21.15.78:0/362520961 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f98340145a0 con 0x7f984014e0b0 2023-10-14T11:59:00.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.309+0000 7f9827fff700 1 --2- 172.21.15.78:0/362520961 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f982c06af00 0x7f982c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:00.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.309+0000 7f9827fff700 1 -- 172.21.15.78:0/362520961 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5232+0+0 (secure 0 0 0) 0x7f9834089b30 con 0x7f984014e0b0 2023-10-14T11:59:00.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.309+0000 7f983e7fc700 1 --2- 172.21.15.78:0/362520961 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f982c06af00 0x7f982c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:00.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.309+0000 7f983e7fc700 1 --2- 172.21.15.78:0/362520961 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f982c06af00 0x7f982c06d3c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f98400aa380 tx=0x7f9830009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:00.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.313+0000 7f9827fff700 1 -- 172.21.15.78:0/362520961 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9834054f10 con 0x7f984014e0b0 2023-10-14T11:59:00.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:00.504+0000 7f98458a5700 1 -- 172.21.15.78:0/362520961 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "testcephfs1_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f98400ac400 con 0x7f984014e0b0 2023-10-14T11:59:01.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.156+0000 7f9827fff700 1 -- 172.21.15.78:0/362520961 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "testcephfs1_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'testcephfs1_data' created v30) v1 ==== 180+0+0 (secure 0 0 0) 0x7f9834022020 con 0x7f984014e0b0 2023-10-14T11:59:01.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.159+0000 7f98458a5700 1 -- 172.21.15.78:0/362520961 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f982c06af00 msgr2=0x7f982c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:01.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.159+0000 7f98458a5700 1 --2- 172.21.15.78:0/362520961 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f982c06af00 0x7f982c06d3c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f98400aa380 tx=0x7f9830009040 comp rx=0 tx=0).stop 2023-10-14T11:59:01.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.159+0000 7f98458a5700 1 -- 172.21.15.78:0/362520961 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f984014e0b0 msgr2=0x7f9840151460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:01.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.159+0000 7f98458a5700 1 --2- 172.21.15.78:0/362520961 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f984014e0b0 0x7f9840151460 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f9834011b00 tx=0x7f9834011ec0 comp rx=0 tx=0).stop 2023-10-14T11:59:01.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.159+0000 7f98458a5700 1 -- 172.21.15.78:0/362520961 shutdown_connections 2023-10-14T11:59:01.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.159+0000 7f98458a5700 1 --2- 172.21.15.78:0/362520961 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f982c06af00 0x7f982c06d3c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:01.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.159+0000 7f98458a5700 1 --2- 172.21.15.78:0/362520961 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f98401e9620 0x7f98401e9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:01.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.159+0000 7f98458a5700 1 --2- 172.21.15.78:0/362520961 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f984014e0b0 0x7f9840151460 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:01.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.159+0000 7f98458a5700 1 --2- 172.21.15.78:0/362520961 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9840148660 0x7f9840148ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:01.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.159+0000 7f98458a5700 1 -- 172.21.15.78:0/362520961 >> 172.21.15.78:0/362520961 conn(0x7f98400c3a50 msgr2=0x7f9840153340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:01.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.159+0000 7f98458a5700 1 -- 172.21.15.78:0/362520961 shutdown_connections 2023-10-14T11:59:01.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.160+0000 7f98458a5700 1 -- 172.21.15.78:0/362520961 wait complete. 2023-10-14T11:59:01.164 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'testcephfs1_data' created 2023-10-14T11:59:01.188 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new testcephfs1 testcephfs1_metadata testcephfs1_data 2023-10-14T11:59:01.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.314+0000 7f8fc1232700 1 Processor -- start 2023-10-14T11:59:01.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.314+0000 7f8fc1232700 1 -- start start 2023-10-14T11:59:01.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.315+0000 7f8fc1232700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8fbc0c35e0 0x7f8fbc1535d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:01.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.315+0000 7f8fc1232700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f8fbc154c20 0x7f8fbc155030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:01.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.315+0000 7f8fc1232700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fbc0c26d0 con 0x7f8fbc14ed10 2023-10-14T11:59:01.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.315+0000 7f8fc1232700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fbc0a8520 con 0x7f8fbc0c35e0 2023-10-14T11:59:01.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.315+0000 7f8fc1232700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fbc0c29b0 con 0x7f8fbc154c20 2023-10-14T11:59:01.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.315+0000 7f8fba59c700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8fbc0c35e0 0x7f8fbc1535d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:01.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.315+0000 7f8fba59c700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8fbc0c35e0 0x7f8fbc1535d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46188/0 (socket says 172.21.15.78:46188) 2023-10-14T11:59:01.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.315+0000 7f8fba59c700 1 -- 172.21.15.78:0/3210488145 learned_addr learned my addr 172.21.15.78:0/3210488145 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:01.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.315+0000 7f8fbb59e700 1 --2- 172.21.15.78:0/3210488145 >> v2:172.21.15.195:3300/0 conn(0x7f8fbc154c20 0x7f8fbc155030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:01.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.315+0000 7f8fba59c700 1 -- 172.21.15.78:0/3210488145 >> v2:172.21.15.195:3300/0 conn(0x7f8fbc154c20 msgr2=0x7f8fbc155030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:01.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.315+0000 7f8fba59c700 1 --2- 172.21.15.78:0/3210488145 >> v2:172.21.15.195:3300/0 conn(0x7f8fbc154c20 0x7f8fbc155030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:01.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.315+0000 7f8fba59c700 1 -- 172.21.15.78:0/3210488145 >> v1:172.21.15.195:6789/0 conn(0x7f8fbc14ed10 legacy=0x7f8fbc152ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:01.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.315+0000 7f8fba59c700 1 -- 172.21.15.78:0/3210488145 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fbc0c2c90 con 0x7f8fbc0c35e0 2023-10-14T11:59:01.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.315+0000 7f8fba59c700 1 --2- 172.21.15.78:0/3210488145 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8fbc0c35e0 0x7f8fbc1535d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc072ed0 tx=0x7f8fac00be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8af2b9185ee364a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:01.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.315+0000 7f8fb9d9b700 1 -- 172.21.15.78:0/3210488145 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fac00dcf0 con 0x7f8fbc0c35e0 2023-10-14T11:59:01.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.315+0000 7f8fb9d9b700 1 -- 172.21.15.78:0/3210488145 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fac00de50 con 0x7f8fbc0c35e0 2023-10-14T11:59:01.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.315+0000 7f8fb9d9b700 1 -- 172.21.15.78:0/3210488145 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fac004250 con 0x7f8fbc0c35e0 2023-10-14T11:59:01.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.316+0000 7f8fc1232700 1 -- 172.21.15.78:0/3210488145 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8fbc0c35e0 msgr2=0x7f8fbc1535d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:01.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.316+0000 7f8fc1232700 1 --2- 172.21.15.78:0/3210488145 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8fbc0c35e0 0x7f8fbc1535d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc072ed0 tx=0x7f8fac00be70 comp rx=0 tx=0).stop 2023-10-14T11:59:01.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.316+0000 7f8fc1232700 1 -- 172.21.15.78:0/3210488145 shutdown_connections 2023-10-14T11:59:01.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.316+0000 7f8fc1232700 1 --2- 172.21.15.78:0/3210488145 >> v2:172.21.15.195:3300/0 conn(0x7f8fbc154c20 0x7f8fbc155030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:01.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.316+0000 7f8fc1232700 1 --2- 172.21.15.78:0/3210488145 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8fbc0c35e0 0x7f8fbc1535d0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:01.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.316+0000 7f8fc1232700 1 -- 172.21.15.78:0/3210488145 >> 172.21.15.78:0/3210488145 conn(0x7f8fbc0c3a50 msgr2=0x7f8fbc0c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:01.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.316+0000 7f8fc1232700 1 -- 172.21.15.78:0/3210488145 shutdown_connections 2023-10-14T11:59:01.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.316+0000 7f8fc1232700 1 -- 172.21.15.78:0/3210488145 wait complete. 2023-10-14T11:59:01.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.318+0000 7f8fc1232700 1 Processor -- start 2023-10-14T11:59:01.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.318+0000 7f8fc1232700 1 -- start start 2023-10-14T11:59:01.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.319+0000 7f8fc1232700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8fbc14ed10 0x7f8fbc14fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:01.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.319+0000 7f8fc1232700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8fbc154c20 0x7f8fbc1ea1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:01.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.319+0000 7f8fc1232700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8fbc0c18a0 0x7f8fbc1ed630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:01.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.319+0000 7f8fc1232700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc1501b0 con 0x7f8fbc0c18a0 2023-10-14T11:59:01.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.319+0000 7f8fc1232700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc1ed1a0 con 0x7f8fbc14ed10 2023-10-14T11:59:01.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.319+0000 7f8fc1232700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc0c13a0 con 0x7f8fbc154c20 2023-10-14T11:59:01.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.319+0000 7f8fbb59e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8fbc0c18a0 0x7f8fbc1ed630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:01.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.319+0000 7f8fbb59e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8fbc0c18a0 0x7f8fbc1ed630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46196/0 (socket says 172.21.15.78:46196) 2023-10-14T11:59:01.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.319+0000 7f8fbb59e700 1 -- 172.21.15.78:0/4020045854 learned_addr learned my addr 172.21.15.78:0/4020045854 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:01.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.319+0000 7f8fba59c700 1 --2- 172.21.15.78:0/4020045854 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8fbc154c20 0x7f8fbc1ea1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:01.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.319+0000 7f8fbad9d700 1 --2- 172.21.15.78:0/4020045854 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8fbc14ed10 0x7f8fbc14fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:01.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.319+0000 7f8fbb59e700 1 -- 172.21.15.78:0/4020045854 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8fbc154c20 msgr2=0x7f8fbc1ea1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:01.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.319+0000 7f8fbb59e700 1 --2- 172.21.15.78:0/4020045854 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8fbc154c20 0x7f8fbc1ea1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:01.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.319+0000 7f8fbb59e700 1 -- 172.21.15.78:0/4020045854 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8fbc14ed10 msgr2=0x7f8fbc14fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:01.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.319+0000 7f8fbb59e700 1 --2- 172.21.15.78:0/4020045854 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8fbc14ed10 0x7f8fbc14fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:01.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.319+0000 7f8fbb59e700 1 -- 172.21.15.78:0/4020045854 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fac00b7f0 con 0x7f8fbc0c18a0 2023-10-14T11:59:01.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.319+0000 7f8fbb59e700 1 --2- 172.21.15.78:0/4020045854 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8fbc0c18a0 0x7f8fbc1ed630 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f8fb400cce0 tx=0x7f8fb400a4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:01.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.319+0000 7f8fa37fe700 1 -- 172.21.15.78:0/4020045854 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb400ce80 con 0x7f8fbc0c18a0 2023-10-14T11:59:01.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.319+0000 7f8fa37fe700 1 -- 172.21.15.78:0/4020045854 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fb400acf0 con 0x7f8fbc0c18a0 2023-10-14T11:59:01.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.319+0000 7f8fc1232700 1 -- 172.21.15.78:0/4020045854 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fbc1ea6f0 con 0x7f8fbc0c18a0 2023-10-14T11:59:01.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.319+0000 7f8fa37fe700 1 -- 172.21.15.78:0/4020045854 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb4003c40 con 0x7f8fbc0c18a0 2023-10-14T11:59:01.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.319+0000 7f8fc1232700 1 -- 172.21.15.78:0/4020045854 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fbc0c0de0 con 0x7f8fbc0c18a0 2023-10-14T11:59:01.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.320+0000 7f8fc1232700 1 -- 172.21.15.78:0/4020045854 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fbc074b80 con 0x7f8fbc0c18a0 2023-10-14T11:59:01.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.321+0000 7f8fa37fe700 1 -- 172.21.15.78:0/4020045854 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8fb400f3f0 con 0x7f8fbc0c18a0 2023-10-14T11:59:01.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.321+0000 7f8fa37fe700 1 --2- 172.21.15.78:0/4020045854 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8fa806af00 0x7f8fa806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:01.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.321+0000 7f8fa37fe700 1 -- 172.21.15.78:0/4020045854 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f8fb4089660 con 0x7f8fbc0c18a0 2023-10-14T11:59:01.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.321+0000 7f8fbad9d700 1 --2- 172.21.15.78:0/4020045854 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8fa806af00 0x7f8fa806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:01.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.322+0000 7f8fbad9d700 1 --2- 172.21.15.78:0/4020045854 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8fa806af00 0x7f8fa806d3c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8fb0001730 tx=0x7f8fb0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:01.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.326+0000 7f8fa37fe700 1 -- 172.21.15.78:0/4020045854 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8fb40547f0 con 0x7f8fbc0c18a0 2023-10-14T11:59:01.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:01.560+0000 7f8fc1232700 1 -- 172.21.15.78:0/4020045854 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "testcephfs1", "metadata": "testcephfs1_metadata", "data": "testcephfs1_data"} v 0) v1 -- 0x7f8fbc1ed2e0 con 0x7f8fbc0c18a0 2023-10-14T11:59:02.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.183+0000 7f8fa37fe700 1 -- 172.21.15.78:0/4020045854 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "testcephfs1", "metadata": "testcephfs1_metadata", "data": "testcephfs1_data"}]=0 new fs with metadata pool 4 and data pool 5 v13) v1 ==== 187+0+0 (secure 0 0 0) 0x7f8fb401d0d0 con 0x7f8fbc0c18a0 2023-10-14T11:59:02.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.186+0000 7f8fc1232700 1 -- 172.21.15.78:0/4020045854 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8fa806af00 msgr2=0x7f8fa806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:02.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.186+0000 7f8fc1232700 1 --2- 172.21.15.78:0/4020045854 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8fa806af00 0x7f8fa806d3c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8fb0001730 tx=0x7f8fb0009040 comp rx=0 tx=0).stop 2023-10-14T11:59:02.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.186+0000 7f8fc1232700 1 -- 172.21.15.78:0/4020045854 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8fbc0c18a0 msgr2=0x7f8fbc1ed630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:02.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.186+0000 7f8fc1232700 1 --2- 172.21.15.78:0/4020045854 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8fbc0c18a0 0x7f8fbc1ed630 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f8fb400cce0 tx=0x7f8fb400a4d0 comp rx=0 tx=0).stop 2023-10-14T11:59:02.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.186+0000 7f8fc1232700 1 -- 172.21.15.78:0/4020045854 shutdown_connections 2023-10-14T11:59:02.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.186+0000 7f8fc1232700 1 --2- 172.21.15.78:0/4020045854 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8fbc0c18a0 0x7f8fbc1ed630 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:02.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.186+0000 7f8fc1232700 1 --2- 172.21.15.78:0/4020045854 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8fa806af00 0x7f8fa806d3c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:02.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.186+0000 7f8fc1232700 1 --2- 172.21.15.78:0/4020045854 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8fbc14ed10 0x7f8fbc14fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:02.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.186+0000 7f8fc1232700 1 --2- 172.21.15.78:0/4020045854 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8fbc154c20 0x7f8fbc1ea1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:02.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.186+0000 7f8fc1232700 1 -- 172.21.15.78:0/4020045854 >> 172.21.15.78:0/4020045854 conn(0x7f8fbc0c3a50 msgr2=0x7f8fbc155380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:02.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.186+0000 7f8fc1232700 1 -- 172.21.15.78:0/4020045854 shutdown_connections 2023-10-14T11:59:02.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.186+0000 7f8fc1232700 1 -- 172.21.15.78:0/4020045854 wait complete. 2023-10-14T11:59:02.191 INFO:teuthology.orchestra.run.smithi078.stderr:new fs with metadata pool 4 and data pool 5 2023-10-14T11:59:02.205 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T11:59:02.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.323+0000 7f09416df700 1 Processor -- start 2023-10-14T11:59:02.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.323+0000 7f09416df700 1 -- start start 2023-10-14T11:59:02.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.323+0000 7f09416df700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f093c1524a0 0x7f093c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:02.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.323+0000 7f09416df700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f093c06d7b0 0x7f093c14ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:02.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.323+0000 7f09416df700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f093c151700 con 0x7f093c0c1ab0 2023-10-14T11:59:02.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.323+0000 7f09416df700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f093c06dbb0 con 0x7f093c1524a0 2023-10-14T11:59:02.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.323+0000 7f09416df700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f093c155ae0 con 0x7f093c06d7b0 2023-10-14T11:59:02.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.324+0000 7f093a7fc700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f093c1524a0 0x7f093c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:02.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.324+0000 7f093a7fc700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f093c1524a0 0x7f093c06d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46202/0 (socket says 172.21.15.78:46202) 2023-10-14T11:59:02.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.324+0000 7f093a7fc700 1 -- 172.21.15.78:0/2832139682 learned_addr learned my addr 172.21.15.78:0/2832139682 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:02.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.324+0000 7f093b7fe700 1 --2- 172.21.15.78:0/2832139682 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f093c06d7b0 0x7f093c14ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:02.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.324+0000 7f093a7fc700 1 -- 172.21.15.78:0/2832139682 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f093c06d7b0 msgr2=0x7f093c14ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:02.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.324+0000 7f093a7fc700 1 --2- 172.21.15.78:0/2832139682 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f093c06d7b0 0x7f093c14ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:02.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.324+0000 7f093a7fc700 1 -- 172.21.15.78:0/2832139682 >> v1:172.21.15.195:6789/0 conn(0x7f093c0c1ab0 legacy=0x7f093c0a8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:02.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.324+0000 7f093a7fc700 1 -- 172.21.15.78:0/2832139682 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f093c1519c0 con 0x7f093c1524a0 2023-10-14T11:59:02.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.324+0000 7f093a7fc700 1 --2- 172.21.15.78:0/2832139682 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f093c1524a0 0x7f093c06d270 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f09300098f0 tx=0x7f0930009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf7c31f7877b7d5c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:02.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.324+0000 7f0939ffb700 1 -- 172.21.15.78:0/2832139682 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0930011830 con 0x7f093c1524a0 2023-10-14T11:59:02.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.324+0000 7f0939ffb700 1 -- 172.21.15.78:0/2832139682 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0930011990 con 0x7f093c1524a0 2023-10-14T11:59:02.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.324+0000 7f0939ffb700 1 -- 172.21.15.78:0/2832139682 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0930011d20 con 0x7f093c1524a0 2023-10-14T11:59:02.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.325+0000 7f09416df700 1 -- 172.21.15.78:0/2832139682 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f093c1524a0 msgr2=0x7f093c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:02.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.325+0000 7f09416df700 1 --2- 172.21.15.78:0/2832139682 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f093c1524a0 0x7f093c06d270 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f09300098f0 tx=0x7f0930009c00 comp rx=0 tx=0).stop 2023-10-14T11:59:02.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.325+0000 7f09416df700 1 -- 172.21.15.78:0/2832139682 shutdown_connections 2023-10-14T11:59:02.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.325+0000 7f09416df700 1 --2- 172.21.15.78:0/2832139682 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f093c06d7b0 0x7f093c14ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:02.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.325+0000 7f09416df700 1 --2- 172.21.15.78:0/2832139682 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f093c1524a0 0x7f093c06d270 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:02.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.325+0000 7f09416df700 1 -- 172.21.15.78:0/2832139682 >> 172.21.15.78:0/2832139682 conn(0x7f093c0c2930 msgr2=0x7f093c0c2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:02.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.325+0000 7f09416df700 1 -- 172.21.15.78:0/2832139682 shutdown_connections 2023-10-14T11:59:02.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.325+0000 7f09416df700 1 -- 172.21.15.78:0/2832139682 wait complete. 2023-10-14T11:59:02.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.326+0000 7f09416df700 1 Processor -- start 2023-10-14T11:59:02.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.326+0000 7f09416df700 1 -- start start 2023-10-14T11:59:02.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.327+0000 7f09416df700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f093c06d7b0 0x7f093c0c5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:02.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.327+0000 7f09416df700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f093c0c1ab0 0x7f093c0bec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:02.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.327+0000 7f09416df700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f093c1524a0 0x7f093c0bf180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:02.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.327+0000 7f09416df700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093c154fd0 con 0x7f093c0c1ab0 2023-10-14T11:59:02.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.327+0000 7f09416df700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093c0bbf40 con 0x7f093c06d7b0 2023-10-14T11:59:02.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.327+0000 7f09416df700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f093c0bf6c0 con 0x7f093c1524a0 2023-10-14T11:59:02.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.327+0000 7f093a7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f093c0c1ab0 0x7f093c0bec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:02.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.327+0000 7f093a7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f093c0c1ab0 0x7f093c0bec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46208/0 (socket says 172.21.15.78:46208) 2023-10-14T11:59:02.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.327+0000 7f093a7fc700 1 -- 172.21.15.78:0/458381437 learned_addr learned my addr 172.21.15.78:0/458381437 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:02.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.327+0000 7f093b7fe700 1 --2- 172.21.15.78:0/458381437 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f093c1524a0 0x7f093c0bf180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:02.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.327+0000 7f093affd700 1 --2- 172.21.15.78:0/458381437 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f093c06d7b0 0x7f093c0c5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:02.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.327+0000 7f093a7fc700 1 -- 172.21.15.78:0/458381437 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f093c1524a0 msgr2=0x7f093c0bf180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:02.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.327+0000 7f093a7fc700 1 --2- 172.21.15.78:0/458381437 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f093c1524a0 0x7f093c0bf180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:02.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.327+0000 7f093a7fc700 1 -- 172.21.15.78:0/458381437 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f093c06d7b0 msgr2=0x7f093c0c5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:02.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.327+0000 7f093a7fc700 1 --2- 172.21.15.78:0/458381437 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f093c06d7b0 0x7f093c0c5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:02.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.327+0000 7f093a7fc700 1 -- 172.21.15.78:0/458381437 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0930009550 con 0x7f093c0c1ab0 2023-10-14T11:59:02.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.327+0000 7f093a7fc700 1 --2- 172.21.15.78:0/458381437 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f093c0c1ab0 0x7f093c0bec40 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f0930009c40 tx=0x7f0930004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:02.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.328+0000 7f09237fe700 1 -- 172.21.15.78:0/458381437 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0930004510 con 0x7f093c0c1ab0 2023-10-14T11:59:02.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.328+0000 7f09237fe700 1 -- 172.21.15.78:0/458381437 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09300036a0 con 0x7f093c0c1ab0 2023-10-14T11:59:02.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.328+0000 7f09416df700 1 -- 172.21.15.78:0/458381437 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f093c154a10 con 0x7f093c0c1ab0 2023-10-14T11:59:02.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.328+0000 7f09237fe700 1 -- 172.21.15.78:0/458381437 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0930003a50 con 0x7f093c0c1ab0 2023-10-14T11:59:02.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.328+0000 7f09416df700 1 -- 172.21.15.78:0/458381437 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f093c0bbbd0 con 0x7f093c0c1ab0 2023-10-14T11:59:02.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.329+0000 7f09416df700 1 -- 172.21.15.78:0/458381437 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f093c0bc0b0 con 0x7f093c0c1ab0 2023-10-14T11:59:02.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.329+0000 7f09237fe700 1 -- 172.21.15.78:0/458381437 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0930003bb0 con 0x7f093c0c1ab0 2023-10-14T11:59:02.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.330+0000 7f09237fe700 1 --2- 172.21.15.78:0/458381437 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f092806af00 0x7f092806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:02.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.330+0000 7f09237fe700 1 -- 172.21.15.78:0/458381437 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f093008c5b0 con 0x7f093c0c1ab0 2023-10-14T11:59:02.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.330+0000 7f093affd700 1 --2- 172.21.15.78:0/458381437 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f092806af00 0x7f092806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:02.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.330+0000 7f093affd700 1 --2- 172.21.15.78:0/458381437 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f092806af00 0x7f092806d3c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f092c007fd0 tx=0x7f092c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:02.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.334+0000 7f09237fe700 1 -- 172.21.15.78:0/458381437 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0930057650 con 0x7f093c0c1ab0 2023-10-14T11:59:02.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.528+0000 7f09416df700 1 -- 172.21.15.78:0/458381437 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f093c074b80 con 0x7f093c0c1ab0 2023-10-14T11:59:02.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.530+0000 7f09237fe700 1 -- 172.21.15.78:0/458381437 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+16578 (secure 0 0 0) 0x7f0930003e60 con 0x7f093c0c1ab0 2023-10-14T11:59:02.532 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:59:02.532 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":31,"fsid":"4c5dc134-3092-406a-a1e1-13e6d9e926e8","created":"2023-10-14T11:57:57.363750+0000","modified":"2023-10-14T11:59:01.562452+0000","last_up_change":"2023-10-14T11:58:06.982472+0000","last_in_change":"2023-10-14T11:57:59.821373+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T11:58:09.001200+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"testcephfs1_metadata","create_time":"2023-10-14T11:58:59.469752+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"testcephfs1"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"testcephfs1_data","create_time":"2023-10-14T11:59:00.521599+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"testcephfs1"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"625a5fc0-9474-4eae-b80e-14c27a74ad19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6801","nonce":3884969780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6803","nonce":3884969780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6807","nonce":3884969780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6805","nonce":3884969780}]},"public_addr":"172.21.15.78:6801/3884969780","cluster_addr":"172.21.15.78:6803/3884969780","heartbeat_back_addr":"172.21.15.78:6807/3884969780","heartbeat_front_addr":"172.21.15.78:6805/3884969780","state":["exists","up"]},{"osd":1,"uuid":"49d63db1-2981-49ab-9c40-c50fc744b982","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6809","nonce":2176928406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6811","nonce":2176928406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6815","nonce":2176928406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6813","nonce":2176928406}]},"public_addr":"172.21.15.78:6809/2176928406","cluster_addr":"172.21.15.78:6811/2176928406","heartbeat_back_addr":"172.21.15.78:6815/2176928406","heartbeat_front_addr":"172.21.15.78:6813/2176928406","state":["exists","up"]},{"osd":2,"uuid":"1d7d0b29-cbd1-4717-9fe6-765a8d4998b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6825","nonce":3647044577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6827","nonce":3647044577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6831","nonce":3647044577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6829","nonce":3647044577}]},"public_addr":"172.21.15.78:6825/3647044577","cluster_addr":"172.21.15.78:6827/3647044577","heartbeat_back_addr":"172.21.15.78:6831/3647044577","heartbeat_front_addr":"172.21.15.78:6829/3647044577","state":["exists","up"]},{"osd":3,"uuid":"2f75ded1-e9e2-40e7-99ae-45e1e425e1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6817","nonce":3584112288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6819","nonce":3584112288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6823","nonce":3584112288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6821","nonce":3584112288}]},"public_addr":"172.21.15.78:6817/3584112288","cluster_addr":"172.21.15.78:6819/3584112288","heartbeat_back_addr":"172.21.15.78:6823/3584112288","heartbeat_front_addr":"172.21.15.78:6821/3584112288","state":["exists","up"]},{"osd":4,"uuid":"5130b432-8358-46cb-8a31-3064c976f6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6801","nonce":3732756301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6803","nonce":3732756301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6807","nonce":3732756301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6805","nonce":3732756301}]},"public_addr":"172.21.15.195:6801/3732756301","cluster_addr":"172.21.15.195:6803/3732756301","heartbeat_back_addr":"172.21.15.195:6807/3732756301","heartbeat_front_addr":"172.21.15.195:6805/3732756301","state":["exists","up"]},{"osd":5,"uuid":"60e234a9-a506-4e40-9c19-49a58f3cf28c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6825","nonce":2857706411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6827","nonce":2857706411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6831","nonce":2857706411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6829","nonce":2857706411}]},"public_addr":"172.21.15.195:6825/2857706411","cluster_addr":"172.21.15.195:6827/2857706411","heartbeat_back_addr":"172.21.15.195:6831/2857706411","heartbeat_front_addr":"172.21.15.195:6829/2857706411","state":["exists","up"]},{"osd":6,"uuid":"df7d2dd6-26b6-4eb1-96e5-94d1f3955b33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6810","nonce":72554374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6813","nonce":72554374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6820","nonce":72554374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6817","nonce":72554374}]},"public_addr":"172.21.15.195:6810/72554374","cluster_addr":"172.21.15.195:6813/72554374","heartbeat_back_addr":"172.21.15.195:6820/72554374","heartbeat_front_addr":"172.21.15.195:6817/72554374","state":["exists","up"]},{"osd":7,"uuid":"47a9d4de-f56d-4aa4-9900-c4a9cf9b2f97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6809","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6811","nonce":482472979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6816","nonce":482472979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6823","nonce":482472979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6821","nonce":482472979}]},"public_addr":"172.21.15.195:6811/482472979","cluster_addr":"172.21.15.195:6816/482472979","heartbeat_back_addr":"172.21.15.195:6823/482472979","heartbeat_front_addr":"172.21.15.195:6821/482472979","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.754003+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.005994+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.043603+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.024506+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.062585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.033767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.994825+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.038300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T11:59:02.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.532+0000 7f09416df700 1 -- 172.21.15.78:0/458381437 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f092806af00 msgr2=0x7f092806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:02.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.532+0000 7f09416df700 1 --2- 172.21.15.78:0/458381437 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f092806af00 0x7f092806d3c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f092c007fd0 tx=0x7f092c009040 comp rx=0 tx=0).stop 2023-10-14T11:59:02.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.532+0000 7f09416df700 1 -- 172.21.15.78:0/458381437 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f093c0c1ab0 msgr2=0x7f093c0bec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:02.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.532+0000 7f09416df700 1 --2- 172.21.15.78:0/458381437 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f093c0c1ab0 0x7f093c0bec40 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f0930009c40 tx=0x7f0930004750 comp rx=0 tx=0).stop 2023-10-14T11:59:02.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.532+0000 7f09416df700 1 -- 172.21.15.78:0/458381437 shutdown_connections 2023-10-14T11:59:02.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.532+0000 7f09416df700 1 --2- 172.21.15.78:0/458381437 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f093c1524a0 0x7f093c0bf180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:02.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.532+0000 7f09416df700 1 --2- 172.21.15.78:0/458381437 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f092806af00 0x7f092806d3c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:02.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.532+0000 7f09416df700 1 --2- 172.21.15.78:0/458381437 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f093c06d7b0 0x7f093c0c5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:02.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.532+0000 7f09416df700 1 --2- 172.21.15.78:0/458381437 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f093c0c1ab0 0x7f093c0bec40 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:02.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.532+0000 7f09416df700 1 -- 172.21.15.78:0/458381437 >> 172.21.15.78:0/458381437 conn(0x7f093c0c2930 msgr2=0x7f093c14bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:02.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.532+0000 7f09416df700 1 -- 172.21.15.78:0/458381437 shutdown_connections 2023-10-14T11:59:02.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.532+0000 7f09416df700 1 -- 172.21.15.78:0/458381437 wait complete. 2023-10-14T11:59:02.545 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T11:59:02.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.663+0000 7fcc9828f700 1 Processor -- start 2023-10-14T11:59:02.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.663+0000 7fcc9828f700 1 -- start start 2023-10-14T11:59:02.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.664+0000 7fcc9828f700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcc9006e360 0x7fcc90156650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:02.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.664+0000 7fcc9828f700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc900c2290 con 0x7fcc900abc90 2023-10-14T11:59:02.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.664+0000 7fcc9828f700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc9006db20 con 0x7fcc90156b90 2023-10-14T11:59:02.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.664+0000 7fcc9828f700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc9006ae70 con 0x7fcc9006e360 2023-10-14T11:59:02.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.664+0000 7fcc9582a700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcc9006e360 0x7fcc90156650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:02.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.664+0000 7fcc9602b700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fcc900abc90 0x7fcc900c1b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51090/0 (socket says 172.21.15.78:51090) 2023-10-14T11:59:02.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.664+0000 7fcc9602b700 1 -- 172.21.15.78:0/4138821522 learned_addr learned my addr 172.21.15.78:0/4138821522 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:59:02.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.664+0000 7fcc9582a700 1 -- 172.21.15.78:0/4138821522 >> v1:172.21.15.195:6789/0 conn(0x7fcc90156b90 legacy=0x7fcc900c2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:02.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.664+0000 7fcc9582a700 1 -- 172.21.15.78:0/4138821522 >> v1:172.21.15.78:6789/0 conn(0x7fcc900abc90 legacy=0x7fcc900c1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:02.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.664+0000 7fcc9582a700 1 -- 172.21.15.78:0/4138821522 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc9006ddc0 con 0x7fcc9006e360 2023-10-14T11:59:02.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.664+0000 7fcc95029700 1 -- 172.21.15.78:0/4138821522 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1429523443 0 0) 0x7fcc900c2290 con 0x7fcc900abc90 2023-10-14T11:59:02.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.664+0000 7fcc9582a700 1 --2- 172.21.15.78:0/4138821522 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc9006e360 0x7fcc90156650 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fcc80009840 tx=0x7fcc80009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6180d26d1aec46c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:02.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.664+0000 7fcc95029700 1 -- 172.21.15.78:0/4138821522 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc80011830 con 0x7fcc9006e360 2023-10-14T11:59:02.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.664+0000 7fcc95029700 1 -- 172.21.15.78:0/4138821522 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc80011990 con 0x7fcc9006e360 2023-10-14T11:59:02.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.664+0000 7fcc95029700 1 -- 172.21.15.78:0/4138821522 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc80011d20 con 0x7fcc9006e360 2023-10-14T11:59:02.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.665+0000 7fcc9828f700 1 -- 172.21.15.78:0/4138821522 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc9006e360 msgr2=0x7fcc90156650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:02.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.665+0000 7fcc9828f700 1 --2- 172.21.15.78:0/4138821522 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc9006e360 0x7fcc90156650 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fcc80009840 tx=0x7fcc80009b50 comp rx=0 tx=0).stop 2023-10-14T11:59:02.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.665+0000 7fcc9828f700 1 -- 172.21.15.78:0/4138821522 shutdown_connections 2023-10-14T11:59:02.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.665+0000 7fcc9828f700 1 --2- 172.21.15.78:0/4138821522 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc9006e360 0x7fcc90156650 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:02.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.665+0000 7fcc9828f700 1 -- 172.21.15.78:0/4138821522 >> 172.21.15.78:0/4138821522 conn(0x7fcc900a8410 msgr2=0x7fcc900a8820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:02.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.665+0000 7fcc9828f700 1 -- 172.21.15.78:0/4138821522 shutdown_connections 2023-10-14T11:59:02.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.665+0000 7fcc9828f700 1 -- 172.21.15.78:0/4138821522 wait complete. 2023-10-14T11:59:02.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.666+0000 7fcc9828f700 1 Processor -- start 2023-10-14T11:59:02.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.666+0000 7fcc9828f700 1 -- start start 2023-10-14T11:59:02.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.667+0000 7fcc9828f700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcc9006e360 0x7fcc900ad740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:02.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.667+0000 7fcc9828f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcc900abc90 0x7fcc901eb150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:02.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.667+0000 7fcc9828f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc90156b90 0x7fcc901eb690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:02.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.667+0000 7fcc9828f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc900c3f10 con 0x7fcc90156b90 2023-10-14T11:59:02.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.667+0000 7fcc9828f700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc900bccf0 con 0x7fcc900abc90 2023-10-14T11:59:02.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.667+0000 7fcc9828f700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc900aee00 con 0x7fcc9006e360 2023-10-14T11:59:02.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.667+0000 7fcc9682c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc90156b90 0x7fcc901eb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:02.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.667+0000 7fcc9682c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc90156b90 0x7fcc901eb690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46224/0 (socket says 172.21.15.78:46224) 2023-10-14T11:59:02.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.667+0000 7fcc9682c700 1 -- 172.21.15.78:0/1274972268 learned_addr learned my addr 172.21.15.78:0/1274972268 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:02.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.667+0000 7fcc9602b700 1 --2- 172.21.15.78:0/1274972268 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcc9006e360 0x7fcc900ad740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:02.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.667+0000 7fcc9582a700 1 --2- 172.21.15.78:0/1274972268 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcc900abc90 0x7fcc901eb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:02.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.667+0000 7fcc9682c700 1 -- 172.21.15.78:0/1274972268 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcc9006e360 msgr2=0x7fcc900ad740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:02.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.667+0000 7fcc9682c700 1 --2- 172.21.15.78:0/1274972268 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcc9006e360 0x7fcc900ad740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:02.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.667+0000 7fcc9682c700 1 -- 172.21.15.78:0/1274972268 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcc900abc90 msgr2=0x7fcc901eb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:02.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.667+0000 7fcc9682c700 1 --2- 172.21.15.78:0/1274972268 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcc900abc90 0x7fcc901eb150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:02.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.667+0000 7fcc9682c700 1 -- 172.21.15.78:0/1274972268 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc800094a0 con 0x7fcc90156b90 2023-10-14T11:59:02.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.667+0000 7fcc9682c700 1 --2- 172.21.15.78:0/1274972268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc90156b90 0x7fcc901eb690 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fcc8800cb80 tx=0x7fcc8800ce90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:02.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.667+0000 7fcc86ffd700 1 -- 172.21.15.78:0/1274972268 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc88003c00 con 0x7fcc90156b90 2023-10-14T11:59:02.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.667+0000 7fcc86ffd700 1 -- 172.21.15.78:0/1274972268 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc88003d60 con 0x7fcc90156b90 2023-10-14T11:59:02.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.667+0000 7fcc9828f700 1 -- 172.21.15.78:0/1274972268 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc900c09a0 con 0x7fcc90156b90 2023-10-14T11:59:02.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.667+0000 7fcc86ffd700 1 -- 172.21.15.78:0/1274972268 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8800aaa0 con 0x7fcc90156b90 2023-10-14T11:59:02.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.668+0000 7fcc9828f700 1 -- 172.21.15.78:0/1274972268 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc901520b0 con 0x7fcc90156b90 2023-10-14T11:59:02.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.668+0000 7fcc9828f700 1 -- 172.21.15.78:0/1274972268 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc900b9770 con 0x7fcc90156b90 2023-10-14T11:59:02.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.669+0000 7fcc86ffd700 1 -- 172.21.15.78:0/1274972268 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fcc8800ac00 con 0x7fcc90156b90 2023-10-14T11:59:02.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.669+0000 7fcc86ffd700 1 --2- 172.21.15.78:0/1274972268 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fcc7406af00 0x7fcc7406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:02.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.669+0000 7fcc9602b700 1 --2- 172.21.15.78:0/1274972268 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fcc7406af00 0x7fcc7406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:02.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.669+0000 7fcc86ffd700 1 -- 172.21.15.78:0/1274972268 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5729+0+0 (secure 0 0 0) 0x7fcc880894a0 con 0x7fcc90156b90 2023-10-14T11:59:02.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.670+0000 7fcc9602b700 1 --2- 172.21.15.78:0/1274972268 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fcc7406af00 0x7fcc7406d3c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c007b10 tx=0x7fcc8c007a60 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:02.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.674+0000 7fcc86ffd700 1 -- 172.21.15.78:0/1274972268 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcc900b9770 con 0x7fcc90156b90 2023-10-14T11:59:02.866 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.864+0000 7fcc9828f700 1 -- 172.21.15.78:0/1274972268 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc900b9770 con 0x7fcc90156b90 2023-10-14T11:59:02.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.865+0000 7fcc86ffd700 1 -- 172.21.15.78:0/1274972268 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+16578 (secure 0 0 0) 0x7fcc900b9770 con 0x7fcc90156b90 2023-10-14T11:59:02.867 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:59:02.868 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":31,"fsid":"4c5dc134-3092-406a-a1e1-13e6d9e926e8","created":"2023-10-14T11:57:57.363750+0000","modified":"2023-10-14T11:59:01.562452+0000","last_up_change":"2023-10-14T11:58:06.982472+0000","last_in_change":"2023-10-14T11:57:59.821373+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T11:58:09.001200+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"testcephfs1_metadata","create_time":"2023-10-14T11:58:59.469752+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"testcephfs1"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"testcephfs1_data","create_time":"2023-10-14T11:59:00.521599+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"testcephfs1"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"625a5fc0-9474-4eae-b80e-14c27a74ad19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6801","nonce":3884969780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6803","nonce":3884969780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6807","nonce":3884969780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6805","nonce":3884969780}]},"public_addr":"172.21.15.78:6801/3884969780","cluster_addr":"172.21.15.78:6803/3884969780","heartbeat_back_addr":"172.21.15.78:6807/3884969780","heartbeat_front_addr":"172.21.15.78:6805/3884969780","state":["exists","up"]},{"osd":1,"uuid":"49d63db1-2981-49ab-9c40-c50fc744b982","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6809","nonce":2176928406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6811","nonce":2176928406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6815","nonce":2176928406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6813","nonce":2176928406}]},"public_addr":"172.21.15.78:6809/2176928406","cluster_addr":"172.21.15.78:6811/2176928406","heartbeat_back_addr":"172.21.15.78:6815/2176928406","heartbeat_front_addr":"172.21.15.78:6813/2176928406","state":["exists","up"]},{"osd":2,"uuid":"1d7d0b29-cbd1-4717-9fe6-765a8d4998b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6825","nonce":3647044577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6827","nonce":3647044577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6831","nonce":3647044577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6829","nonce":3647044577}]},"public_addr":"172.21.15.78:6825/3647044577","cluster_addr":"172.21.15.78:6827/3647044577","heartbeat_back_addr":"172.21.15.78:6831/3647044577","heartbeat_front_addr":"172.21.15.78:6829/3647044577","state":["exists","up"]},{"osd":3,"uuid":"2f75ded1-e9e2-40e7-99ae-45e1e425e1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6817","nonce":3584112288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6819","nonce":3584112288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6823","nonce":3584112288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6821","nonce":3584112288}]},"public_addr":"172.21.15.78:6817/3584112288","cluster_addr":"172.21.15.78:6819/3584112288","heartbeat_back_addr":"172.21.15.78:6823/3584112288","heartbeat_front_addr":"172.21.15.78:6821/3584112288","state":["exists","up"]},{"osd":4,"uuid":"5130b432-8358-46cb-8a31-3064c976f6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6801","nonce":3732756301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6803","nonce":3732756301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6807","nonce":3732756301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6805","nonce":3732756301}]},"public_addr":"172.21.15.195:6801/3732756301","cluster_addr":"172.21.15.195:6803/3732756301","heartbeat_back_addr":"172.21.15.195:6807/3732756301","heartbeat_front_addr":"172.21.15.195:6805/3732756301","state":["exists","up"]},{"osd":5,"uuid":"60e234a9-a506-4e40-9c19-49a58f3cf28c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6825","nonce":2857706411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6827","nonce":2857706411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6831","nonce":2857706411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6829","nonce":2857706411}]},"public_addr":"172.21.15.195:6825/2857706411","cluster_addr":"172.21.15.195:6827/2857706411","heartbeat_back_addr":"172.21.15.195:6831/2857706411","heartbeat_front_addr":"172.21.15.195:6829/2857706411","state":["exists","up"]},{"osd":6,"uuid":"df7d2dd6-26b6-4eb1-96e5-94d1f3955b33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6810","nonce":72554374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6813","nonce":72554374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6820","nonce":72554374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6817","nonce":72554374}]},"public_addr":"172.21.15.195:6810/72554374","cluster_addr":"172.21.15.195:6813/72554374","heartbeat_back_addr":"172.21.15.195:6820/72554374","heartbeat_front_addr":"172.21.15.195:6817/72554374","state":["exists","up"]},{"osd":7,"uuid":"47a9d4de-f56d-4aa4-9900-c4a9cf9b2f97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6809","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6811","nonce":482472979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6816","nonce":482472979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6823","nonce":482472979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6821","nonce":482472979}]},"public_addr":"172.21.15.195:6811/482472979","cluster_addr":"172.21.15.195:6816/482472979","heartbeat_back_addr":"172.21.15.195:6823/482472979","heartbeat_front_addr":"172.21.15.195:6821/482472979","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.754003+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.005994+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.043603+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.024506+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.062585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.033767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.994825+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.038300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T11:59:02.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.867+0000 7fcc9828f700 1 -- 172.21.15.78:0/1274972268 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fcc7406af00 msgr2=0x7fcc7406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:02.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.867+0000 7fcc9828f700 1 --2- 172.21.15.78:0/1274972268 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fcc7406af00 0x7fcc7406d3c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c007b10 tx=0x7fcc8c007a60 comp rx=0 tx=0).stop 2023-10-14T11:59:02.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.867+0000 7fcc9828f700 1 -- 172.21.15.78:0/1274972268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc90156b90 msgr2=0x7fcc901eb690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:02.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.867+0000 7fcc9828f700 1 --2- 172.21.15.78:0/1274972268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc90156b90 0x7fcc901eb690 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fcc8800cb80 tx=0x7fcc8800ce90 comp rx=0 tx=0).stop 2023-10-14T11:59:02.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.867+0000 7fcc9828f700 1 -- 172.21.15.78:0/1274972268 shutdown_connections 2023-10-14T11:59:02.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.867+0000 7fcc9828f700 1 --2- 172.21.15.78:0/1274972268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc90156b90 0x7fcc901eb690 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:02.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.867+0000 7fcc9828f700 1 --2- 172.21.15.78:0/1274972268 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcc9006e360 0x7fcc900ad740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:02.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.867+0000 7fcc9828f700 1 --2- 172.21.15.78:0/1274972268 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fcc7406af00 0x7fcc7406d3c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:02.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.867+0000 7fcc9828f700 1 --2- 172.21.15.78:0/1274972268 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcc900abc90 0x7fcc901eb150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:02.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.867+0000 7fcc9828f700 1 -- 172.21.15.78:0/1274972268 >> 172.21.15.78:0/1274972268 conn(0x7fcc900a8410 msgr2=0x7fcc900c2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:02.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.867+0000 7fcc9828f700 1 -- 172.21.15.78:0/1274972268 shutdown_connections 2023-10-14T11:59:02.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:02.867+0000 7fcc9828f700 1 -- 172.21.15.78:0/1274972268 wait complete. 2023-10-14T11:59:02.882 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set testcephfs1 standby_count_wanted 0 2023-10-14T11:59:03.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.004+0000 7f7d0089e700 1 Processor -- start 2023-10-14T11:59:03.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.004+0000 7f7d0089e700 1 -- start start 2023-10-14T11:59:03.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.005+0000 7f7d0089e700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f7cfc0c6cd0 0x7f7cfc0c70b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:03.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.005+0000 7f7d0089e700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7cfc14ebd0 0x7f7cfc1581d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:03.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.005+0000 7f7d0089e700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cfc0c4c40 con 0x7f7cfc0bd950 2023-10-14T11:59:03.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.005+0000 7f7d0089e700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cfc158e20 con 0x7f7cfc14ebd0 2023-10-14T11:59:03.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.005+0000 7f7d0089e700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cfc0c54e0 con 0x7f7cfc0c6cd0 2023-10-14T11:59:03.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.005+0000 7f7cf9d9b700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7cfc14ebd0 0x7f7cfc1581d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:03.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.005+0000 7f7cf9d9b700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7cfc14ebd0 0x7f7cfc1581d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46230/0 (socket says 172.21.15.78:46230) 2023-10-14T11:59:03.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.005+0000 7f7cf9d9b700 1 -- 172.21.15.78:0/3525831760 learned_addr learned my addr 172.21.15.78:0/3525831760 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:03.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.005+0000 7f7cfa59c700 1 --2- 172.21.15.78:0/3525831760 >> v2:172.21.15.195:3300/0 conn(0x7f7cfc0c6cd0 0x7f7cfc0c70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:03.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.005+0000 7f7cf9d9b700 1 -- 172.21.15.78:0/3525831760 >> v2:172.21.15.195:3300/0 conn(0x7f7cfc0c6cd0 msgr2=0x7f7cfc0c70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:03.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.005+0000 7f7cf9d9b700 1 --2- 172.21.15.78:0/3525831760 >> v2:172.21.15.195:3300/0 conn(0x7f7cfc0c6cd0 0x7f7cfc0c70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:03.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.005+0000 7f7cf9d9b700 1 -- 172.21.15.78:0/3525831760 >> v1:172.21.15.78:6789/0 conn(0x7f7cfc0bd950 legacy=0x7f7cfc158710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:03.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.005+0000 7f7cf9d9b700 1 -- 172.21.15.78:0/3525831760 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cfc1556d0 con 0x7f7cfc14ebd0 2023-10-14T11:59:03.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.005+0000 7f7cf959a700 1 -- 172.21.15.78:0/3525831760 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4042095444 0 0) 0x7f7cfc0c4c40 con 0x7f7cfc0bd950 2023-10-14T11:59:03.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.006+0000 7f7cf9d9b700 1 --2- 172.21.15.78:0/3525831760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cfc14ebd0 0x7f7cfc1581d0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc06c9e0 tx=0x7f7cec009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebd224c63393b743 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:03.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.006+0000 7f7cf959a700 1 -- 172.21.15.78:0/3525831760 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec00b870 con 0x7f7cfc14ebd0 2023-10-14T11:59:03.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.006+0000 7f7cf959a700 1 -- 172.21.15.78:0/3525831760 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cec00b9d0 con 0x7f7cfc14ebd0 2023-10-14T11:59:03.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.006+0000 7f7cf959a700 1 -- 172.21.15.78:0/3525831760 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec00bd60 con 0x7f7cfc14ebd0 2023-10-14T11:59:03.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.006+0000 7f7d0089e700 1 -- 172.21.15.78:0/3525831760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cfc14ebd0 msgr2=0x7f7cfc1581d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:03.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.006+0000 7f7d0089e700 1 --2- 172.21.15.78:0/3525831760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cfc14ebd0 0x7f7cfc1581d0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc06c9e0 tx=0x7f7cec009bd0 comp rx=0 tx=0).stop 2023-10-14T11:59:03.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.006+0000 7f7d0089e700 1 -- 172.21.15.78:0/3525831760 shutdown_connections 2023-10-14T11:59:03.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.006+0000 7f7d0089e700 1 --2- 172.21.15.78:0/3525831760 >> v2:172.21.15.195:3300/0 conn(0x7f7cfc0c6cd0 0x7f7cfc0c70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:03.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.006+0000 7f7d0089e700 1 --2- 172.21.15.78:0/3525831760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cfc14ebd0 0x7f7cfc1581d0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:03.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.006+0000 7f7d0089e700 1 -- 172.21.15.78:0/3525831760 >> 172.21.15.78:0/3525831760 conn(0x7f7cfc0abdf0 msgr2=0x7f7cfc0ac1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:03.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.006+0000 7f7d0089e700 1 -- 172.21.15.78:0/3525831760 shutdown_connections 2023-10-14T11:59:03.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.006+0000 7f7d0089e700 1 -- 172.21.15.78:0/3525831760 wait complete. 2023-10-14T11:59:03.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.008+0000 7f7d0089e700 1 Processor -- start 2023-10-14T11:59:03.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.008+0000 7f7d0089e700 1 -- start start 2023-10-14T11:59:03.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.008+0000 7f7d0089e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7cfc0bd950 0x7f7cfc157a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:03.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.008+0000 7f7d0089e700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7cfc0c6cd0 0x7f7cfc0c3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:03.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.008+0000 7f7d0089e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cfc148c90 0x7f7cfc156e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:03.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.008+0000 7f7d0089e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc1ed230 con 0x7f7cfc148c90 2023-10-14T11:59:03.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.008+0000 7f7d0089e700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc1ed510 con 0x7f7cfc0bd950 2023-10-14T11:59:03.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.008+0000 7f7d0089e700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc1ed7f0 con 0x7f7cfc0c6cd0 2023-10-14T11:59:03.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.008+0000 7f7cfad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cfc148c90 0x7f7cfc156e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:03.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.008+0000 7f7cfad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cfc148c90 0x7f7cfc156e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46238/0 (socket says 172.21.15.78:46238) 2023-10-14T11:59:03.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.008+0000 7f7cfad9d700 1 -- 172.21.15.78:0/4242221866 learned_addr learned my addr 172.21.15.78:0/4242221866 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:03.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.009+0000 7f7cfa59c700 1 --2- 172.21.15.78:0/4242221866 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7cfc0bd950 0x7f7cfc157a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:03.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.009+0000 7f7cf9d9b700 1 --2- 172.21.15.78:0/4242221866 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7cfc0c6cd0 0x7f7cfc0c3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:03.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.009+0000 7f7cfad9d700 1 -- 172.21.15.78:0/4242221866 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7cfc0c6cd0 msgr2=0x7f7cfc0c3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:03.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.009+0000 7f7cfad9d700 1 --2- 172.21.15.78:0/4242221866 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7cfc0c6cd0 0x7f7cfc0c3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:03.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.009+0000 7f7cfad9d700 1 -- 172.21.15.78:0/4242221866 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7cfc0bd950 msgr2=0x7f7cfc157a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:03.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.009+0000 7f7cfad9d700 1 --2- 172.21.15.78:0/4242221866 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7cfc0bd950 0x7f7cfc157a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:03.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.009+0000 7f7cfad9d700 1 -- 172.21.15.78:0/4242221866 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cec009550 con 0x7f7cfc148c90 2023-10-14T11:59:03.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.009+0000 7f7cf9d9b700 1 --2- 172.21.15.78:0/4242221866 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7cfc0c6cd0 0x7f7cfc0c3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:59:03.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.009+0000 7f7cfa59c700 1 --2- 172.21.15.78:0/4242221866 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7cfc0bd950 0x7f7cfc157a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:59:03.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.009+0000 7f7cfad9d700 1 --2- 172.21.15.78:0/4242221866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cfc148c90 0x7f7cfc156e40 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4001560 tx=0x7f7cf400cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:03.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.009+0000 7f7cf2ffd700 1 -- 172.21.15.78:0/4242221866 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf4003e30 con 0x7f7cfc148c90 2023-10-14T11:59:03.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.009+0000 7f7cf2ffd700 1 -- 172.21.15.78:0/4242221866 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cf4005070 con 0x7f7cfc148c90 2023-10-14T11:59:03.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.009+0000 7f7d0089e700 1 -- 172.21.15.78:0/4242221866 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cfc1edad0 con 0x7f7cfc148c90 2023-10-14T11:59:03.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.009+0000 7f7cf2ffd700 1 -- 172.21.15.78:0/4242221866 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf4005420 con 0x7f7cfc148c90 2023-10-14T11:59:03.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.009+0000 7f7d0089e700 1 -- 172.21.15.78:0/4242221866 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cfc0c43e0 con 0x7f7cfc148c90 2023-10-14T11:59:03.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.010+0000 7f7d0089e700 1 -- 172.21.15.78:0/4242221866 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cfc1e9d10 con 0x7f7cfc148c90 2023-10-14T11:59:03.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.010+0000 7f7cf2ffd700 1 -- 172.21.15.78:0/4242221866 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7cf4005580 con 0x7f7cfc148c90 2023-10-14T11:59:03.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.011+0000 7f7cf2ffd700 1 --2- 172.21.15.78:0/4242221866 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7cdc06af00 0x7f7cdc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:03.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.011+0000 7f7cfa59c700 1 --2- 172.21.15.78:0/4242221866 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7cdc06af00 0x7f7cdc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:03.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.011+0000 7f7cf2ffd700 1 -- 172.21.15.78:0/4242221866 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f7cf4088cf0 con 0x7f7cfc148c90 2023-10-14T11:59:03.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.011+0000 7f7cfa59c700 1 --2- 172.21.15.78:0/4242221866 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7cdc06af00 0x7f7cdc06d3c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7ce40099f0 tx=0x7f7ce4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:03.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.015+0000 7f7cf2ffd700 1 -- 172.21.15.78:0/4242221866 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7cf408d070 con 0x7f7cfc148c90 2023-10-14T11:59:03.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:03.251+0000 7f7d0089e700 1 -- 172.21.15.78:0/4242221866 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "testcephfs1", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f7cfc0683e0 con 0x7f7cfc148c90 2023-10-14T11:59:04.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.197+0000 7f7cf2ffd700 1 -- 172.21.15.78:0/4242221866 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "testcephfs1", "var": "standby_count_wanted", "val": "0"}]=0 v16) v1 ==== 123+0+0 (secure 0 0 0) 0x7f7cf4017020 con 0x7f7cfc148c90 2023-10-14T11:59:04.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.199+0000 7f7d0089e700 1 -- 172.21.15.78:0/4242221866 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7cdc06af00 msgr2=0x7f7cdc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:04.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.199+0000 7f7d0089e700 1 --2- 172.21.15.78:0/4242221866 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7cdc06af00 0x7f7cdc06d3c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7ce40099f0 tx=0x7f7ce4008040 comp rx=0 tx=0).stop 2023-10-14T11:59:04.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.199+0000 7f7d0089e700 1 -- 172.21.15.78:0/4242221866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cfc148c90 msgr2=0x7f7cfc156e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:04.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.200+0000 7f7d0089e700 1 --2- 172.21.15.78:0/4242221866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cfc148c90 0x7f7cfc156e40 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4001560 tx=0x7f7cf400cc00 comp rx=0 tx=0).stop 2023-10-14T11:59:04.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.200+0000 7f7d0089e700 1 -- 172.21.15.78:0/4242221866 shutdown_connections 2023-10-14T11:59:04.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.200+0000 7f7d0089e700 1 --2- 172.21.15.78:0/4242221866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cfc148c90 0x7f7cfc156e40 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:04.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.200+0000 7f7d0089e700 1 --2- 172.21.15.78:0/4242221866 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7cdc06af00 0x7f7cdc06d3c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:04.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.200+0000 7f7d0089e700 1 --2- 172.21.15.78:0/4242221866 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7cfc0bd950 0x7f7cfc157a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:04.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.200+0000 7f7d0089e700 1 --2- 172.21.15.78:0/4242221866 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7cfc0c6cd0 0x7f7cfc0c3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:04.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.200+0000 7f7d0089e700 1 -- 172.21.15.78:0/4242221866 >> 172.21.15.78:0/4242221866 conn(0x7f7cfc0abdf0 msgr2=0x7f7cfc06b610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:04.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.200+0000 7f7d0089e700 1 -- 172.21.15.78:0/4242221866 shutdown_connections 2023-10-14T11:59:04.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.200+0000 7f7d0089e700 1 -- 172.21.15.78:0/4242221866 wait complete. 2023-10-14T11:59:04.216 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-14T11:59:04.216 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set testcephfs1 allow_standby_replay false 2023-10-14T11:59:04.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.345+0000 7fbdf093b700 1 Processor -- start 2023-10-14T11:59:04.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.345+0000 7fbdf093b700 1 -- start start 2023-10-14T11:59:04.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.346+0000 7fbdf093b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fbdec06ab40 0x7fbdec0c3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:04.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.346+0000 7fbdf093b700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdec0c39d0 con 0x7fbdec072640 2023-10-14T11:59:04.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.346+0000 7fbdf093b700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdec0c3c00 con 0x7fbdec155f80 2023-10-14T11:59:04.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.346+0000 7fbdf093b700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdec0c0bc0 con 0x7fbdec06ab40 2023-10-14T11:59:04.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.346+0000 7fbde9d9b700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fbdec072640 0x7fbdec14fee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51118/0 (socket says 172.21.15.78:51118) 2023-10-14T11:59:04.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.346+0000 7fbde9d9b700 1 -- 172.21.15.78:0/151570493 learned_addr learned my addr 172.21.15.78:0/151570493 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:59:04.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.347+0000 7fbdea59c700 1 --2- 172.21.15.78:0/151570493 >> v2:172.21.15.195:3300/0 conn(0x7fbdec06ab40 0x7fbdec0c3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:04.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.347+0000 7fbde959a700 1 -- 172.21.15.78:0/151570493 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4030378948 0 0) 0x7fbdec0c39d0 con 0x7fbdec072640 2023-10-14T11:59:04.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.347+0000 7fbde959a700 1 -- 172.21.15.78:0/151570493 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdd40035b0 con 0x7fbdec072640 2023-10-14T11:59:04.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.347+0000 7fbde959a700 1 -- 172.21.15.78:0/151570493 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3947558612 0 0) 0x7fbdd40035b0 con 0x7fbdec072640 2023-10-14T11:59:04.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.347+0000 7fbde959a700 1 -- 172.21.15.78:0/151570493 >> v2:172.21.15.195:3300/0 conn(0x7fbdec06ab40 msgr2=0x7fbdec0c3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:04.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.347+0000 7fbde959a700 1 --2- 172.21.15.78:0/151570493 >> v2:172.21.15.195:3300/0 conn(0x7fbdec06ab40 0x7fbdec0c3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:04.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.347+0000 7fbde959a700 1 -- 172.21.15.78:0/151570493 >> v1:172.21.15.195:6789/0 conn(0x7fbdec155f80 legacy=0x7fbdec1563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:04.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.347+0000 7fbde959a700 1 -- 172.21.15.78:0/151570493 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdec0c25e0 con 0x7fbdec072640 2023-10-14T11:59:04.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.347+0000 7fbde959a700 1 -- 172.21.15.78:0/151570493 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441799917 0 0) 0x7fbddc001670 con 0x7fbdec072640 2023-10-14T11:59:04.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.347+0000 7fbde959a700 1 -- 172.21.15.78:0/151570493 >> v1:172.21.15.78:6789/0 conn(0x7fbdec072640 legacy=0x7fbdec14fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:04.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.347+0000 7fbde959a700 1 --2- 172.21.15.78:0/151570493 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbdd4003a00 0x7fbdd4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:04.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.347+0000 7fbde959a700 1 --2- 172.21.15.78:0/151570493 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbdd4004850 0x7fbdd4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:04.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.347+0000 7fbde959a700 1 --2- 172.21.15.78:0/151570493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbdd4005210 0x7fbdd400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:04.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.347+0000 7fbde959a700 1 -- 172.21.15.78:0/151570493 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec0c0bc0 con 0x7fbdd4005210 2023-10-14T11:59:04.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.347+0000 7fbde959a700 1 -- 172.21.15.78:0/151570493 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd400c0b0 con 0x7fbdd4003a00 2023-10-14T11:59:04.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.347+0000 7fbde959a700 1 -- 172.21.15.78:0/151570493 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd400c270 con 0x7fbdd4004850 2023-10-14T11:59:04.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.347+0000 7fbdead9d700 1 --2- 172.21.15.78:0/151570493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbdd4005210 0x7fbdd400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:04.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.347+0000 7fbdea59c700 1 --2- 172.21.15.78:0/151570493 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbdd4003a00 0x7fbdd4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:04.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.347+0000 7fbde9d9b700 1 --2- 172.21.15.78:0/151570493 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbdd4004850 0x7fbdd4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:04.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.347+0000 7fbdead9d700 1 -- 172.21.15.78:0/151570493 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbdd4004850 msgr2=0x7fbdd4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:04.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.347+0000 7fbdead9d700 1 --2- 172.21.15.78:0/151570493 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbdd4004850 0x7fbdd4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:04.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.347+0000 7fbdead9d700 1 -- 172.21.15.78:0/151570493 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbdd4003a00 msgr2=0x7fbdd4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:04.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.347+0000 7fbdead9d700 1 --2- 172.21.15.78:0/151570493 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbdd4003a00 0x7fbdd4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:04.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.347+0000 7fbdead9d700 1 -- 172.21.15.78:0/151570493 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd40035b0 con 0x7fbdd4005210 2023-10-14T11:59:04.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.347+0000 7fbde9d9b700 1 --2- 172.21.15.78:0/151570493 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbdd4004850 0x7fbdd4004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:59:04.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.347+0000 7fbdead9d700 1 -- 172.21.15.78:0/151570493 reap_dead start 2023-10-14T11:59:04.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.348+0000 7fbdead9d700 1 --2- 172.21.15.78:0/151570493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbdd4005210 0x7fbdd400ba10 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fbde400c240 tx=0x7fbdd4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e45d294b2f130d0f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:04.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.348+0000 7fbde959a700 1 -- 172.21.15.78:0/151570493 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde4001df0 con 0x7fbdd4005210 2023-10-14T11:59:04.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.348+0000 7fbde959a700 1 -- 172.21.15.78:0/151570493 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde400cda0 con 0x7fbdd4005210 2023-10-14T11:59:04.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.348+0000 7fbde959a700 1 -- 172.21.15.78:0/151570493 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde4003c40 con 0x7fbdd4005210 2023-10-14T11:59:04.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.348+0000 7fbdf093b700 1 -- 172.21.15.78:0/151570493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbdd4005210 msgr2=0x7fbdd400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:04.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.348+0000 7fbdf093b700 1 --2- 172.21.15.78:0/151570493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbdd4005210 0x7fbdd400ba10 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fbde400c240 tx=0x7fbdd4005880 comp rx=0 tx=0).stop 2023-10-14T11:59:04.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.348+0000 7fbdf093b700 1 -- 172.21.15.78:0/151570493 shutdown_connections 2023-10-14T11:59:04.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.348+0000 7fbdf093b700 1 --2- 172.21.15.78:0/151570493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbdd4005210 0x7fbdd400ba10 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:04.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.348+0000 7fbdf093b700 1 -- 172.21.15.78:0/151570493 >> 172.21.15.78:0/151570493 conn(0x7fbdec0a84a0 msgr2=0x7fbdec0a88b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:04.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.348+0000 7fbdf093b700 1 -- 172.21.15.78:0/151570493 shutdown_connections 2023-10-14T11:59:04.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.348+0000 7fbdf093b700 1 -- 172.21.15.78:0/151570493 wait complete. 2023-10-14T11:59:04.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.350+0000 7fbdf093b700 1 Processor -- start 2023-10-14T11:59:04.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.350+0000 7fbdf093b700 1 -- start start 2023-10-14T11:59:04.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.350+0000 7fbdf093b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbdd4005210 0x7fbdec1dd650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:04.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.350+0000 7fbdf093b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbdec14adf0 0x7fbdec144740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:04.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.350+0000 7fbdf093b700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbdec148a70 0x7fbdec148f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:04.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.350+0000 7fbdf093b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec149440 con 0x7fbdd4005210 2023-10-14T11:59:04.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.350+0000 7fbdf093b700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec0be650 con 0x7fbdec14adf0 2023-10-14T11:59:04.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.350+0000 7fbdf093b700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec153a10 con 0x7fbdec148a70 2023-10-14T11:59:04.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.350+0000 7fbdea59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbdd4005210 0x7fbdec1dd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:04.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.350+0000 7fbdea59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbdd4005210 0x7fbdec1dd650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46254/0 (socket says 172.21.15.78:46254) 2023-10-14T11:59:04.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.350+0000 7fbdea59c700 1 -- 172.21.15.78:0/951162836 learned_addr learned my addr 172.21.15.78:0/951162836 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:04.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.351+0000 7fbdead9d700 1 --2- 172.21.15.78:0/951162836 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbdec148a70 0x7fbdec148f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:04.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.351+0000 7fbde9d9b700 1 --2- 172.21.15.78:0/951162836 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbdec14adf0 0x7fbdec144740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:04.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.351+0000 7fbdea59c700 1 -- 172.21.15.78:0/951162836 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbdec148a70 msgr2=0x7fbdec148f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:04.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.351+0000 7fbdea59c700 1 --2- 172.21.15.78:0/951162836 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbdec148a70 0x7fbdec148f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:04.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.351+0000 7fbdea59c700 1 -- 172.21.15.78:0/951162836 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbdec14adf0 msgr2=0x7fbdec144740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:04.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.351+0000 7fbdea59c700 1 --2- 172.21.15.78:0/951162836 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbdec14adf0 0x7fbdec144740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:04.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.351+0000 7fbdea59c700 1 -- 172.21.15.78:0/951162836 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdec0be400 con 0x7fbdd4005210 2023-10-14T11:59:04.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.351+0000 7fbdea59c700 1 --2- 172.21.15.78:0/951162836 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbdd4005210 0x7fbdec1dd650 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fbde00119d0 tx=0x7fbde0011ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:04.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.351+0000 7fbddaffd700 1 -- 172.21.15.78:0/951162836 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde0011e70 con 0x7fbdd4005210 2023-10-14T11:59:04.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.351+0000 7fbddaffd700 1 -- 172.21.15.78:0/951162836 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde000f940 con 0x7fbdd4005210 2023-10-14T11:59:04.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.351+0000 7fbdf093b700 1 -- 172.21.15.78:0/951162836 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdec1dcbb0 con 0x7fbdd4005210 2023-10-14T11:59:04.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.351+0000 7fbddaffd700 1 -- 172.21.15.78:0/951162836 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde000fcd0 con 0x7fbdd4005210 2023-10-14T11:59:04.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.351+0000 7fbdf093b700 1 -- 172.21.15.78:0/951162836 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdec1dce50 con 0x7fbdd4005210 2023-10-14T11:59:04.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.352+0000 7fbdf093b700 1 -- 172.21.15.78:0/951162836 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdec074b80 con 0x7fbdd4005210 2023-10-14T11:59:04.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.352+0000 7fbddaffd700 1 -- 172.21.15.78:0/951162836 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fbde0014440 con 0x7fbdd4005210 2023-10-14T11:59:04.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.353+0000 7fbddaffd700 1 --2- 172.21.15.78:0/951162836 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fbdcc06af00 0x7fbdcc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:04.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.353+0000 7fbddaffd700 1 -- 172.21.15.78:0/951162836 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7fbde0089ba0 con 0x7fbdd4005210 2023-10-14T11:59:04.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.353+0000 7fbde9d9b700 1 --2- 172.21.15.78:0/951162836 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fbdcc06af00 0x7fbdcc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:04.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.353+0000 7fbde9d9b700 1 --2- 172.21.15.78:0/951162836 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fbdcc06af00 0x7fbdcc06d3c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbdec14a890 tx=0x7fbddc007040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:04.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.357+0000 7fbddaffd700 1 -- 172.21.15.78:0/951162836 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbde0054d70 con 0x7fbdd4005210 2023-10-14T11:59:04.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:04.591+0000 7fbdf093b700 1 -- 172.21.15.78:0/951162836 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "testcephfs1", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fbdec0a8c70 con 0x7fbdd4005210 2023-10-14T11:59:05.150 INFO:tasks.ceph.mds.d.smithi195.stderr:2023-10-14T11:59:05.148+0000 7f343474b700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-14T11:59:05.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.201+0000 7fbddaffd700 1 -- 172.21.15.78:0/951162836 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "testcephfs1", "var": "allow_standby_replay", "val": "false"}]=0 v17) v1 ==== 127+0+0 (secure 0 0 0) 0x7fbde0058390 con 0x7fbdd4005210 2023-10-14T11:59:05.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.203+0000 7fbdf093b700 1 -- 172.21.15.78:0/951162836 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fbdcc06af00 msgr2=0x7fbdcc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:05.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.203+0000 7fbdf093b700 1 --2- 172.21.15.78:0/951162836 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fbdcc06af00 0x7fbdcc06d3c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbdec14a890 tx=0x7fbddc007040 comp rx=0 tx=0).stop 2023-10-14T11:59:05.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.203+0000 7fbdf093b700 1 -- 172.21.15.78:0/951162836 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbdd4005210 msgr2=0x7fbdec1dd650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:05.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.203+0000 7fbdf093b700 1 --2- 172.21.15.78:0/951162836 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbdd4005210 0x7fbdec1dd650 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fbde00119d0 tx=0x7fbde0011ce0 comp rx=0 tx=0).stop 2023-10-14T11:59:05.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.204+0000 7fbdf093b700 1 -- 172.21.15.78:0/951162836 shutdown_connections 2023-10-14T11:59:05.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.204+0000 7fbdf093b700 1 --2- 172.21.15.78:0/951162836 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbdec148a70 0x7fbdec148f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:05.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.204+0000 7fbdf093b700 1 --2- 172.21.15.78:0/951162836 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbdd4005210 0x7fbdec1dd650 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:05.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.204+0000 7fbdf093b700 1 --2- 172.21.15.78:0/951162836 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fbdcc06af00 0x7fbdcc06d3c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:05.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.204+0000 7fbdf093b700 1 --2- 172.21.15.78:0/951162836 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbdec14adf0 0x7fbdec144740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:05.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.204+0000 7fbdf093b700 1 -- 172.21.15.78:0/951162836 >> 172.21.15.78:0/951162836 conn(0x7fbdec0a84a0 msgr2=0x7fbdec154720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:05.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.204+0000 7fbdf093b700 1 -- 172.21.15.78:0/951162836 shutdown_connections 2023-10-14T11:59:05.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.204+0000 7fbdf093b700 1 -- 172.21.15.78:0/951162836 wait complete. 2023-10-14T11:59:05.219 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'testcephfs1' 2023-10-14T11:59:05.219 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T11:59:05.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.340+0000 7fef8116e700 1 Processor -- start 2023-10-14T11:59:05.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.340+0000 7fef8116e700 1 -- start start 2023-10-14T11:59:05.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.341+0000 7fef8116e700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fef7c0c44c0 0x7fef7c0c7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:05.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.341+0000 7fef8116e700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fef7c0c2c90 0x7fef7c0c3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:05.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.341+0000 7fef8116e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef7c156b20 con 0x7fef7c0c72a0 2023-10-14T11:59:05.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.341+0000 7fef8116e700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef7c155f10 con 0x7fef7c0c2c90 2023-10-14T11:59:05.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.341+0000 7fef8116e700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef7c1561f0 con 0x7fef7c0c44c0 2023-10-14T11:59:05.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.341+0000 7fef7b59e700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fef7c0c2c90 0x7fef7c0c3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:05.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.341+0000 7fef7b59e700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fef7c0c2c90 0x7fef7c0c3100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46266/0 (socket says 172.21.15.78:46266) 2023-10-14T11:59:05.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.341+0000 7fef7b59e700 1 -- 172.21.15.78:0/1439493859 learned_addr learned my addr 172.21.15.78:0/1439493859 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:05.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.341+0000 7fef7ad9d700 1 --2- 172.21.15.78:0/1439493859 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fef7c0c44c0 0x7fef7c0c7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:05.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.341+0000 7fef7b59e700 1 -- 172.21.15.78:0/1439493859 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fef7c0c44c0 msgr2=0x7fef7c0c7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:05.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.341+0000 7fef7b59e700 1 --2- 172.21.15.78:0/1439493859 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fef7c0c44c0 0x7fef7c0c7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:05.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.341+0000 7fef7b59e700 1 -- 172.21.15.78:0/1439493859 >> v1:172.21.15.195:6789/0 conn(0x7fef7c0c72a0 legacy=0x7fef7c0c86e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:05.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.341+0000 7fef7b59e700 1 -- 172.21.15.78:0/1439493859 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef7c1564d0 con 0x7fef7c0c2c90 2023-10-14T11:59:05.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.341+0000 7fef7ad9d700 1 --2- 172.21.15.78:0/1439493859 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fef7c0c44c0 0x7fef7c0c7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:59:05.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.342+0000 7fef7b59e700 1 --2- 172.21.15.78:0/1439493859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef7c0c2c90 0x7fef7c0c3100 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fef74009e50 tx=0x7fef7400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca4f07c8ebd77b1c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:05.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.342+0000 7fef79d9b700 1 -- 172.21.15.78:0/1439493859 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef7400e070 con 0x7fef7c0c2c90 2023-10-14T11:59:05.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.342+0000 7fef79d9b700 1 -- 172.21.15.78:0/1439493859 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef74003a20 con 0x7fef7c0c2c90 2023-10-14T11:59:05.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.342+0000 7fef79d9b700 1 -- 172.21.15.78:0/1439493859 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef74003dd0 con 0x7fef7c0c2c90 2023-10-14T11:59:05.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.342+0000 7fef8116e700 1 -- 172.21.15.78:0/1439493859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef7c0c2c90 msgr2=0x7fef7c0c3100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:05.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.342+0000 7fef8116e700 1 --2- 172.21.15.78:0/1439493859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef7c0c2c90 0x7fef7c0c3100 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fef74009e50 tx=0x7fef7400b330 comp rx=0 tx=0).stop 2023-10-14T11:59:05.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.342+0000 7fef8116e700 1 -- 172.21.15.78:0/1439493859 shutdown_connections 2023-10-14T11:59:05.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.342+0000 7fef8116e700 1 --2- 172.21.15.78:0/1439493859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef7c0c2c90 0x7fef7c0c3100 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:05.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.342+0000 7fef8116e700 1 --2- 172.21.15.78:0/1439493859 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fef7c0c44c0 0x7fef7c0c7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:05.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.342+0000 7fef8116e700 1 -- 172.21.15.78:0/1439493859 >> 172.21.15.78:0/1439493859 conn(0x7fef7c0737d0 msgr2=0x7fef7c073be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:05.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.342+0000 7fef8116e700 1 -- 172.21.15.78:0/1439493859 shutdown_connections 2023-10-14T11:59:05.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.342+0000 7fef8116e700 1 -- 172.21.15.78:0/1439493859 wait complete. 2023-10-14T11:59:05.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.344+0000 7fef8116e700 1 Processor -- start 2023-10-14T11:59:05.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.344+0000 7fef8116e700 1 -- start start 2023-10-14T11:59:05.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.345+0000 7fef8116e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fef7c0c2c90 0x7fef7c1ea290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:05.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.345+0000 7fef8116e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef7c0c44c0 0x7fef7c1ea7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:05.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.345+0000 7fef8116e700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fef7c0c72a0 0x7fef7c0c0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:05.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.345+0000 7fef8116e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef7c14ee90 con 0x7fef7c0c44c0 2023-10-14T11:59:05.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.345+0000 7fef8116e700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef7c1e8ce0 con 0x7fef7c0c2c90 2023-10-14T11:59:05.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.345+0000 7fef8116e700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef7c14f780 con 0x7fef7c0c72a0 2023-10-14T11:59:05.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.345+0000 7fef7a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef7c0c44c0 0x7fef7c1ea7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:05.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.345+0000 7fef7a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef7c0c44c0 0x7fef7c1ea7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46270/0 (socket says 172.21.15.78:46270) 2023-10-14T11:59:05.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.345+0000 7fef7a59c700 1 -- 172.21.15.78:0/1323399879 learned_addr learned my addr 172.21.15.78:0/1323399879 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:05.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.345+0000 7fef7ad9d700 1 --2- 172.21.15.78:0/1323399879 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fef7c0c2c90 0x7fef7c1ea290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:05.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.345+0000 7fef7b59e700 1 --2- 172.21.15.78:0/1323399879 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fef7c0c72a0 0x7fef7c0c0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:05.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.345+0000 7fef7a59c700 1 -- 172.21.15.78:0/1323399879 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fef7c0c72a0 msgr2=0x7fef7c0c0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:05.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.345+0000 7fef7a59c700 1 --2- 172.21.15.78:0/1323399879 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fef7c0c72a0 0x7fef7c0c0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:05.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.345+0000 7fef7a59c700 1 -- 172.21.15.78:0/1323399879 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fef7c0c2c90 msgr2=0x7fef7c1ea290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:05.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.345+0000 7fef7a59c700 1 --2- 172.21.15.78:0/1323399879 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fef7c0c2c90 0x7fef7c1ea290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:05.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.346+0000 7fef7a59c700 1 -- 172.21.15.78:0/1323399879 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef74009b30 con 0x7fef7c0c44c0 2023-10-14T11:59:05.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.346+0000 7fef7ad9d700 1 --2- 172.21.15.78:0/1323399879 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fef7c0c2c90 0x7fef7c1ea290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:59:05.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.346+0000 7fef7a59c700 1 --2- 172.21.15.78:0/1323399879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef7c0c44c0 0x7fef7c1ea7d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fef6c00b4a0 tx=0x7fef6c00b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:05.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.346+0000 7fef637fe700 1 -- 172.21.15.78:0/1323399879 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6c011830 con 0x7fef7c0c44c0 2023-10-14T11:59:05.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.346+0000 7fef637fe700 1 -- 172.21.15.78:0/1323399879 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef6c011990 con 0x7fef7c0c44c0 2023-10-14T11:59:05.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.346+0000 7fef8116e700 1 -- 172.21.15.78:0/1323399879 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef7c0ab600 con 0x7fef7c0c44c0 2023-10-14T11:59:05.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.346+0000 7fef637fe700 1 -- 172.21.15.78:0/1323399879 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6c011d40 con 0x7fef7c0c44c0 2023-10-14T11:59:05.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.346+0000 7fef8116e700 1 -- 172.21.15.78:0/1323399879 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef7c0ab040 con 0x7fef7c0c44c0 2023-10-14T11:59:05.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.347+0000 7fef8116e700 1 -- 172.21.15.78:0/1323399879 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef7c0aaea0 con 0x7fef7c0c44c0 2023-10-14T11:59:05.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.347+0000 7fef637fe700 1 -- 172.21.15.78:0/1323399879 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fef6c01a400 con 0x7fef7c0c44c0 2023-10-14T11:59:05.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.348+0000 7fef637fe700 1 --2- 172.21.15.78:0/1323399879 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fef6806af00 0x7fef6806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:05.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.348+0000 7fef637fe700 1 -- 172.21.15.78:0/1323399879 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7fef6c089200 con 0x7fef7c0c44c0 2023-10-14T11:59:05.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.348+0000 7fef7ad9d700 1 --2- 172.21.15.78:0/1323399879 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fef6806af00 0x7fef6806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:05.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.348+0000 7fef7ad9d700 1 --2- 172.21.15.78:0/1323399879 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fef6806af00 0x7fef6806d3c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fef70006fd0 tx=0x7fef70008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:05.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.352+0000 7fef637fe700 1 -- 172.21.15.78:0/1323399879 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fef7c0aaea0 con 0x7fef7c0c44c0 2023-10-14T11:59:05.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.584+0000 7fef8116e700 1 -- 172.21.15.78:0/1323399879 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef7c0aaea0 con 0x7fef7c0c44c0 2023-10-14T11:59:05.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.585+0000 7fef637fe700 1 -- 172.21.15.78:0/1323399879 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4924 (secure 0 0 0) 0x7fef7c0aaea0 con 0x7fef7c0c44c0 2023-10-14T11:59:05.587 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:59:05.588 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4658,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6834/1492486786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1492486786},{"type":"v1","addr":"172.21.15.195:6834","nonce":1492486786}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5016,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/2991384297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2991384297},{"type":"v1","addr":"172.21.15.78:6836","nonce":2991384297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5019,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3246640037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3246640037},{"type":"v1","addr":"172.21.15.78:6837","nonce":3246640037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T11:59:01.562764+0000","modified":"2023-10-14T11:59:04.592935+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5031":{"gid":5031,"name":"d","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.195:6835/2130671508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6833","nonce":2130671508},{"type":"v1","addr":"172.21.15.195:6835","nonce":2130671508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"testcephfs1","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-14T11:59:05.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.587+0000 7fef8116e700 1 -- 172.21.15.78:0/1323399879 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fef6806af00 msgr2=0x7fef6806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:05.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.587+0000 7fef8116e700 1 --2- 172.21.15.78:0/1323399879 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fef6806af00 0x7fef6806d3c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fef70006fd0 tx=0x7fef70008040 comp rx=0 tx=0).stop 2023-10-14T11:59:05.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.587+0000 7fef8116e700 1 -- 172.21.15.78:0/1323399879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef7c0c44c0 msgr2=0x7fef7c1ea7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:05.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.587+0000 7fef8116e700 1 --2- 172.21.15.78:0/1323399879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef7c0c44c0 0x7fef7c1ea7d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fef6c00b4a0 tx=0x7fef6c00b860 comp rx=0 tx=0).stop 2023-10-14T11:59:05.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.587+0000 7fef8116e700 1 -- 172.21.15.78:0/1323399879 shutdown_connections 2023-10-14T11:59:05.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.587+0000 7fef8116e700 1 --2- 172.21.15.78:0/1323399879 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fef7c0c72a0 0x7fef7c0c0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:05.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.587+0000 7fef8116e700 1 --2- 172.21.15.78:0/1323399879 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fef6806af00 0x7fef6806d3c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:05.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.587+0000 7fef8116e700 1 --2- 172.21.15.78:0/1323399879 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fef7c0c2c90 0x7fef7c1ea290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:05.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.587+0000 7fef8116e700 1 --2- 172.21.15.78:0/1323399879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef7c0c44c0 0x7fef7c1ea7d0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:05.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.587+0000 7fef8116e700 1 -- 172.21.15.78:0/1323399879 >> 172.21.15.78:0/1323399879 conn(0x7fef7c0737d0 msgr2=0x7fef7c0668f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:05.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.587+0000 7fef8116e700 1 -- 172.21.15.78:0/1323399879 shutdown_connections 2023-10-14T11:59:05.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.587+0000 7fef8116e700 1 -- 172.21.15.78:0/1323399879 wait complete. 2023-10-14T11:59:05.592 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 17 2023-10-14T11:59:05.601 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T11:59:05.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.719+0000 7fe63997f700 1 Processor -- start 2023-10-14T11:59:05.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.719+0000 7fe63997f700 1 -- start start 2023-10-14T11:59:05.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.720+0000 7fe63997f700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe63406ab40 0x7fe63406f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:05.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.720+0000 7fe63997f700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe6341554e0 0x7fe63406af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:05.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.720+0000 7fe63997f700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6340c49f0 con 0x7fe6340c0350 2023-10-14T11:59:05.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.720+0000 7fe63997f700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6340726a0 con 0x7fe6341554e0 2023-10-14T11:59:05.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.720+0000 7fe63997f700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe634072980 con 0x7fe63406ab40 2023-10-14T11:59:05.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.720+0000 7fe6327fc700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe6341554e0 0x7fe63406af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:05.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.720+0000 7fe6337fe700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fe6340c0350 0x7fe63413d7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51156/0 (socket says 172.21.15.78:51156) 2023-10-14T11:59:05.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.720+0000 7fe6337fe700 1 -- 172.21.15.78:0/4156618589 learned_addr learned my addr 172.21.15.78:0/4156618589 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:59:05.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.720+0000 7fe632ffd700 1 --2- 172.21.15.78:0/4156618589 >> v2:172.21.15.195:3300/0 conn(0x7fe63406ab40 0x7fe63406f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:05.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.720+0000 7fe6327fc700 1 -- 172.21.15.78:0/4156618589 >> v2:172.21.15.195:3300/0 conn(0x7fe63406ab40 msgr2=0x7fe63406f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:05.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.720+0000 7fe6327fc700 1 --2- 172.21.15.78:0/4156618589 >> v2:172.21.15.195:3300/0 conn(0x7fe63406ab40 0x7fe63406f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:05.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.720+0000 7fe6327fc700 1 -- 172.21.15.78:0/4156618589 >> v1:172.21.15.78:6789/0 conn(0x7fe6340c0350 legacy=0x7fe63413d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:05.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.720+0000 7fe6327fc700 1 -- 172.21.15.78:0/4156618589 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe634154a00 con 0x7fe6341554e0 2023-10-14T11:59:05.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.720+0000 7fe631ffb700 1 -- 172.21.15.78:0/4156618589 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1892127177 0 0) 0x7fe6340c49f0 con 0x7fe6340c0350 2023-10-14T11:59:05.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.721+0000 7fe6327fc700 1 --2- 172.21.15.78:0/4156618589 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6341554e0 0x7fe63406af20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe634072bc0 tx=0x7fe624009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfa5eb2eed526e12 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:05.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.721+0000 7fe631ffb700 1 -- 172.21.15.78:0/4156618589 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe624011830 con 0x7fe6341554e0 2023-10-14T11:59:05.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.721+0000 7fe631ffb700 1 -- 172.21.15.78:0/4156618589 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe624011990 con 0x7fe6341554e0 2023-10-14T11:59:05.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.721+0000 7fe631ffb700 1 -- 172.21.15.78:0/4156618589 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe624011d20 con 0x7fe6341554e0 2023-10-14T11:59:05.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.721+0000 7fe63997f700 1 -- 172.21.15.78:0/4156618589 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6341554e0 msgr2=0x7fe63406af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:05.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.721+0000 7fe63997f700 1 --2- 172.21.15.78:0/4156618589 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6341554e0 0x7fe63406af20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe634072bc0 tx=0x7fe624009b20 comp rx=0 tx=0).stop 2023-10-14T11:59:05.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.721+0000 7fe63997f700 1 -- 172.21.15.78:0/4156618589 shutdown_connections 2023-10-14T11:59:05.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.721+0000 7fe63997f700 1 --2- 172.21.15.78:0/4156618589 >> v2:172.21.15.195:3300/0 conn(0x7fe63406ab40 0x7fe63406f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:05.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.721+0000 7fe63997f700 1 --2- 172.21.15.78:0/4156618589 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6341554e0 0x7fe63406af20 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:05.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.721+0000 7fe63997f700 1 -- 172.21.15.78:0/4156618589 >> 172.21.15.78:0/4156618589 conn(0x7fe6340c3ac0 msgr2=0x7fe6340c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:05.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.721+0000 7fe63997f700 1 -- 172.21.15.78:0/4156618589 shutdown_connections 2023-10-14T11:59:05.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.721+0000 7fe63997f700 1 -- 172.21.15.78:0/4156618589 wait complete. 2023-10-14T11:59:05.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.722+0000 7fe63997f700 1 Processor -- start 2023-10-14T11:59:05.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.723+0000 7fe63997f700 1 -- start start 2023-10-14T11:59:05.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.723+0000 7fe63997f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe63406ab40 0x7fe634133970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:05.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.723+0000 7fe63997f700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe6340c0350 0x7fe6341ed680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:05.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.723+0000 7fe63997f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe6341554e0 0x7fe6341edbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:05.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.723+0000 7fe63997f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6341ee1c0 con 0x7fe63406ab40 2023-10-14T11:59:05.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.723+0000 7fe63997f700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6341e9ec0 con 0x7fe6341554e0 2023-10-14T11:59:05.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.723+0000 7fe63997f700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6341ea1a0 con 0x7fe6340c0350 2023-10-14T11:59:05.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.723+0000 7fe632ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe63406ab40 0x7fe634133970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:05.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.723+0000 7fe632ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe63406ab40 0x7fe634133970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46282/0 (socket says 172.21.15.78:46282) 2023-10-14T11:59:05.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.723+0000 7fe632ffd700 1 -- 172.21.15.78:0/1087764665 learned_addr learned my addr 172.21.15.78:0/1087764665 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:05.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.724+0000 7fe632ffd700 1 -- 172.21.15.78:0/1087764665 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe6340c0350 msgr2=0x7fe6341ed680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:05.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.724+0000 7fe6327fc700 1 --2- 172.21.15.78:0/1087764665 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe6340c0350 0x7fe6341ed680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:05.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.724+0000 7fe6337fe700 1 --2- 172.21.15.78:0/1087764665 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe6341554e0 0x7fe6341edbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:05.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.724+0000 7fe632ffd700 1 --2- 172.21.15.78:0/1087764665 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe6340c0350 0x7fe6341ed680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:05.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.724+0000 7fe632ffd700 1 -- 172.21.15.78:0/1087764665 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe6341554e0 msgr2=0x7fe6341edbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:05.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.724+0000 7fe632ffd700 1 --2- 172.21.15.78:0/1087764665 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe6341554e0 0x7fe6341edbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:05.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.724+0000 7fe632ffd700 1 -- 172.21.15.78:0/1087764665 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6240094a0 con 0x7fe63406ab40 2023-10-14T11:59:05.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.724+0000 7fe632ffd700 1 --2- 172.21.15.78:0/1087764665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe63406ab40 0x7fe634133970 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe62800e810 tx=0x7fe62800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:05.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.724+0000 7fe61bfff700 1 -- 172.21.15.78:0/1087764665 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe628013610 con 0x7fe63406ab40 2023-10-14T11:59:05.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.724+0000 7fe61bfff700 1 -- 172.21.15.78:0/1087764665 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe628013770 con 0x7fe63406ab40 2023-10-14T11:59:05.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.724+0000 7fe63997f700 1 -- 172.21.15.78:0/1087764665 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6341ea480 con 0x7fe63406ab40 2023-10-14T11:59:05.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.724+0000 7fe61bfff700 1 -- 172.21.15.78:0/1087764665 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe628013b20 con 0x7fe63406ab40 2023-10-14T11:59:05.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.724+0000 7fe63997f700 1 -- 172.21.15.78:0/1087764665 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6341eac80 con 0x7fe63406ab40 2023-10-14T11:59:05.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.725+0000 7fe63997f700 1 -- 172.21.15.78:0/1087764665 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe634074b80 con 0x7fe63406ab40 2023-10-14T11:59:05.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.726+0000 7fe61bfff700 1 -- 172.21.15.78:0/1087764665 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe628013c80 con 0x7fe63406ab40 2023-10-14T11:59:05.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.726+0000 7fe61bfff700 1 --2- 172.21.15.78:0/1087764665 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe62006af00 0x7fe62006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:05.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.726+0000 7fe61bfff700 1 -- 172.21.15.78:0/1087764665 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7fe628089770 con 0x7fe63406ab40 2023-10-14T11:59:05.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.726+0000 7fe6327fc700 1 --2- 172.21.15.78:0/1087764665 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe62006af00 0x7fe62006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:05.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.727+0000 7fe6327fc700 1 --2- 172.21.15.78:0/1087764665 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe62006af00 0x7fe62006d3c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe6340c3010 tx=0x7fe624008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:05.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.730+0000 7fe61bfff700 1 -- 172.21.15.78:0/1087764665 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe6280548e0 con 0x7fe63406ab40 2023-10-14T11:59:05.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.919+0000 7fe63997f700 1 -- 172.21.15.78:0/1087764665 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe63414eb50 con 0x7fe63406ab40 2023-10-14T11:59:05.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.920+0000 7fe61bfff700 1 -- 172.21.15.78:0/1087764665 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+16565 (secure 0 0 0) 0x7fe628057f00 con 0x7fe63406ab40 2023-10-14T11:59:05.922 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:59:05.923 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":32,"fsid":"4c5dc134-3092-406a-a1e1-13e6d9e926e8","created":"2023-10-14T11:57:57.363750+0000","modified":"2023-10-14T11:59:03.127845+0000","last_up_change":"2023-10-14T11:58:06.982472+0000","last_in_change":"2023-10-14T11:57:59.821373+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T11:58:09.001200+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"testcephfs1_metadata","create_time":"2023-10-14T11:58:59.469752+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"testcephfs1"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"testcephfs1_data","create_time":"2023-10-14T11:59:00.521599+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"testcephfs1"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"625a5fc0-9474-4eae-b80e-14c27a74ad19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6801","nonce":3884969780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6803","nonce":3884969780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6807","nonce":3884969780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6805","nonce":3884969780}]},"public_addr":"172.21.15.78:6801/3884969780","cluster_addr":"172.21.15.78:6803/3884969780","heartbeat_back_addr":"172.21.15.78:6807/3884969780","heartbeat_front_addr":"172.21.15.78:6805/3884969780","state":["exists","up"]},{"osd":1,"uuid":"49d63db1-2981-49ab-9c40-c50fc744b982","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6809","nonce":2176928406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6811","nonce":2176928406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6815","nonce":2176928406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6813","nonce":2176928406}]},"public_addr":"172.21.15.78:6809/2176928406","cluster_addr":"172.21.15.78:6811/2176928406","heartbeat_back_addr":"172.21.15.78:6815/2176928406","heartbeat_front_addr":"172.21.15.78:6813/2176928406","state":["exists","up"]},{"osd":2,"uuid":"1d7d0b29-cbd1-4717-9fe6-765a8d4998b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6825","nonce":3647044577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6827","nonce":3647044577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6831","nonce":3647044577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6829","nonce":3647044577}]},"public_addr":"172.21.15.78:6825/3647044577","cluster_addr":"172.21.15.78:6827/3647044577","heartbeat_back_addr":"172.21.15.78:6831/3647044577","heartbeat_front_addr":"172.21.15.78:6829/3647044577","state":["exists","up"]},{"osd":3,"uuid":"2f75ded1-e9e2-40e7-99ae-45e1e425e1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6817","nonce":3584112288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6819","nonce":3584112288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6823","nonce":3584112288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6821","nonce":3584112288}]},"public_addr":"172.21.15.78:6817/3584112288","cluster_addr":"172.21.15.78:6819/3584112288","heartbeat_back_addr":"172.21.15.78:6823/3584112288","heartbeat_front_addr":"172.21.15.78:6821/3584112288","state":["exists","up"]},{"osd":4,"uuid":"5130b432-8358-46cb-8a31-3064c976f6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6801","nonce":3732756301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6803","nonce":3732756301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6807","nonce":3732756301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6805","nonce":3732756301}]},"public_addr":"172.21.15.195:6801/3732756301","cluster_addr":"172.21.15.195:6803/3732756301","heartbeat_back_addr":"172.21.15.195:6807/3732756301","heartbeat_front_addr":"172.21.15.195:6805/3732756301","state":["exists","up"]},{"osd":5,"uuid":"60e234a9-a506-4e40-9c19-49a58f3cf28c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6825","nonce":2857706411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6827","nonce":2857706411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6831","nonce":2857706411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6829","nonce":2857706411}]},"public_addr":"172.21.15.195:6825/2857706411","cluster_addr":"172.21.15.195:6827/2857706411","heartbeat_back_addr":"172.21.15.195:6831/2857706411","heartbeat_front_addr":"172.21.15.195:6829/2857706411","state":["exists","up"]},{"osd":6,"uuid":"df7d2dd6-26b6-4eb1-96e5-94d1f3955b33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6810","nonce":72554374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6813","nonce":72554374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6820","nonce":72554374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6817","nonce":72554374}]},"public_addr":"172.21.15.195:6810/72554374","cluster_addr":"172.21.15.195:6813/72554374","heartbeat_back_addr":"172.21.15.195:6820/72554374","heartbeat_front_addr":"172.21.15.195:6817/72554374","state":["exists","up"]},{"osd":7,"uuid":"47a9d4de-f56d-4aa4-9900-c4a9cf9b2f97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6809","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6811","nonce":482472979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6816","nonce":482472979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6823","nonce":482472979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6821","nonce":482472979}]},"public_addr":"172.21.15.195:6811/482472979","cluster_addr":"172.21.15.195:6816/482472979","heartbeat_back_addr":"172.21.15.195:6823/482472979","heartbeat_front_addr":"172.21.15.195:6821/482472979","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.754003+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.005994+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.043603+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.024506+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.062585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.033767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.994825+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.038300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T11:59:05.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.922+0000 7fe63997f700 1 -- 172.21.15.78:0/1087764665 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe62006af00 msgr2=0x7fe62006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:05.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.922+0000 7fe63997f700 1 --2- 172.21.15.78:0/1087764665 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe62006af00 0x7fe62006d3c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe6340c3010 tx=0x7fe624008030 comp rx=0 tx=0).stop 2023-10-14T11:59:05.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.922+0000 7fe63997f700 1 -- 172.21.15.78:0/1087764665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe63406ab40 msgr2=0x7fe634133970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:05.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.922+0000 7fe63997f700 1 --2- 172.21.15.78:0/1087764665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe63406ab40 0x7fe634133970 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe62800e810 tx=0x7fe62800ebd0 comp rx=0 tx=0).stop 2023-10-14T11:59:05.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.922+0000 7fe63997f700 1 -- 172.21.15.78:0/1087764665 shutdown_connections 2023-10-14T11:59:05.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.922+0000 7fe63997f700 1 --2- 172.21.15.78:0/1087764665 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe62006af00 0x7fe62006d3c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:05.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.922+0000 7fe63997f700 1 --2- 172.21.15.78:0/1087764665 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe6341554e0 0x7fe6341edbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:05.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.922+0000 7fe63997f700 1 --2- 172.21.15.78:0/1087764665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe63406ab40 0x7fe634133970 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:05.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.922+0000 7fe63997f700 1 --2- 172.21.15.78:0/1087764665 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe6340c0350 0x7fe6341ed680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:05.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.923+0000 7fe63997f700 1 -- 172.21.15.78:0/1087764665 >> 172.21.15.78:0/1087764665 conn(0x7fe6340c3ac0 msgr2=0x7fe6340667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:05.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.923+0000 7fe63997f700 1 -- 172.21.15.78:0/1087764665 shutdown_connections 2023-10-14T11:59:05.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:05.923+0000 7fe63997f700 1 -- 172.21.15.78:0/1087764665 wait complete. 2023-10-14T11:59:05.937 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-14T11:59:05.937 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T11:59:06.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.055+0000 7f1c215c3700 1 Processor -- start 2023-10-14T11:59:06.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.055+0000 7f1c215c3700 1 -- start start 2023-10-14T11:59:06.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.056+0000 7f1c215c3700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1c1c14b7e0 0x7f1c1c0c5150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:06.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.056+0000 7f1c215c3700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1c1c154b90 0x7f1c1c0c9260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:06.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.056+0000 7f1c215c3700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c1c0c4e40 con 0x7f1c1c0c97a0 2023-10-14T11:59:06.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.056+0000 7f1c215c3700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c1c0c81c0 con 0x7f1c1c154b90 2023-10-14T11:59:06.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.056+0000 7f1c215c3700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1c0c84a0 con 0x7f1c1c14b7e0 2023-10-14T11:59:06.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.056+0000 7f1c1a7fc700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1c1c154b90 0x7f1c1c0c9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:06.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.056+0000 7f1c1a7fc700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1c1c154b90 0x7f1c1c0c9260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46294/0 (socket says 172.21.15.78:46294) 2023-10-14T11:59:06.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.056+0000 7f1c1a7fc700 1 -- 172.21.15.78:0/761665820 learned_addr learned my addr 172.21.15.78:0/761665820 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:06.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.056+0000 7f1c1affd700 1 --2- 172.21.15.78:0/761665820 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1c1c14b7e0 0x7f1c1c0c5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:06.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.056+0000 7f1c1a7fc700 1 -- 172.21.15.78:0/761665820 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1c1c14b7e0 msgr2=0x7f1c1c0c5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:06.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.056+0000 7f1c1a7fc700 1 --2- 172.21.15.78:0/761665820 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1c1c14b7e0 0x7f1c1c0c5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:06.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.056+0000 7f1c1a7fc700 1 -- 172.21.15.78:0/761665820 >> v1:172.21.15.195:6789/0 conn(0x7f1c1c0c97a0 legacy=0x7f1c1c06b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:06.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.056+0000 7f1c1a7fc700 1 -- 172.21.15.78:0/761665820 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c1c0c8780 con 0x7f1c1c154b90 2023-10-14T11:59:06.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.056+0000 7f1c1a7fc700 1 --2- 172.21.15.78:0/761665820 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c1c154b90 0x7f1c1c0c9260 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c0098f0 tx=0x7f1c0c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54efebfb531d46ec server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:06.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.057+0000 7f1c19ffb700 1 -- 172.21.15.78:0/761665820 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c011830 con 0x7f1c1c154b90 2023-10-14T11:59:06.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.057+0000 7f1c19ffb700 1 -- 172.21.15.78:0/761665820 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c0c011990 con 0x7f1c1c154b90 2023-10-14T11:59:06.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.057+0000 7f1c19ffb700 1 -- 172.21.15.78:0/761665820 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c011d20 con 0x7f1c1c154b90 2023-10-14T11:59:06.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.057+0000 7f1c215c3700 1 -- 172.21.15.78:0/761665820 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c1c154b90 msgr2=0x7f1c1c0c9260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:06.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.057+0000 7f1c215c3700 1 --2- 172.21.15.78:0/761665820 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c1c154b90 0x7f1c1c0c9260 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c0098f0 tx=0x7f1c0c009c00 comp rx=0 tx=0).stop 2023-10-14T11:59:06.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.057+0000 7f1c215c3700 1 -- 172.21.15.78:0/761665820 shutdown_connections 2023-10-14T11:59:06.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.057+0000 7f1c215c3700 1 --2- 172.21.15.78:0/761665820 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1c1c14b7e0 0x7f1c1c0c5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:06.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.057+0000 7f1c215c3700 1 --2- 172.21.15.78:0/761665820 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c1c154b90 0x7f1c1c0c9260 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:06.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.057+0000 7f1c215c3700 1 -- 172.21.15.78:0/761665820 >> 172.21.15.78:0/761665820 conn(0x7f1c1c0ca2c0 msgr2=0x7f1c1c0ca6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:06.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.057+0000 7f1c215c3700 1 -- 172.21.15.78:0/761665820 shutdown_connections 2023-10-14T11:59:06.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.057+0000 7f1c215c3700 1 -- 172.21.15.78:0/761665820 wait complete. 2023-10-14T11:59:06.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.058+0000 7f1c215c3700 1 Processor -- start 2023-10-14T11:59:06.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.059+0000 7f1c215c3700 1 -- start start 2023-10-14T11:59:06.069 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.059+0000 7f1c215c3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c1c14b7e0 0x7f1c1c158800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:06.069 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.059+0000 7f1c215c3700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1c1c154b90 0x7f1c1c157d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:06.069 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.059+0000 7f1c215c3700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1c1c0ab9e0 0x7f1c1c0c0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:06.069 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.059+0000 7f1c215c3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1c0beeb0 con 0x7f1c1c14b7e0 2023-10-14T11:59:06.069 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.059+0000 7f1c215c3700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1c158ec0 con 0x7f1c1c0ab9e0 2023-10-14T11:59:06.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.059+0000 7f1c215c3700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1c0bf150 con 0x7f1c1c154b90 2023-10-14T11:59:06.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.059+0000 7f1c1affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c1c14b7e0 0x7f1c1c158800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:06.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.059+0000 7f1c1affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c1c14b7e0 0x7f1c1c158800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46296/0 (socket says 172.21.15.78:46296) 2023-10-14T11:59:06.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.059+0000 7f1c1affd700 1 -- 172.21.15.78:0/3060168872 learned_addr learned my addr 172.21.15.78:0/3060168872 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:06.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.059+0000 7f1c1a7fc700 1 --2- 172.21.15.78:0/3060168872 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1c1c154b90 0x7f1c1c157d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:06.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.059+0000 7f1c1b7fe700 1 --2- 172.21.15.78:0/3060168872 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1c1c0ab9e0 0x7f1c1c0c0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:06.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.059+0000 7f1c1affd700 1 -- 172.21.15.78:0/3060168872 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1c1c154b90 msgr2=0x7f1c1c157d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:06.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.060+0000 7f1c1affd700 1 --2- 172.21.15.78:0/3060168872 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1c1c154b90 0x7f1c1c157d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:06.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.060+0000 7f1c1affd700 1 -- 172.21.15.78:0/3060168872 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1c1c0ab9e0 msgr2=0x7f1c1c0c0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:06.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.060+0000 7f1c1affd700 1 --2- 172.21.15.78:0/3060168872 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1c1c0ab9e0 0x7f1c1c0c0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:06.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.060+0000 7f1c1affd700 1 -- 172.21.15.78:0/3060168872 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c0c009550 con 0x7f1c1c14b7e0 2023-10-14T11:59:06.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.060+0000 7f1c1b7fe700 1 --2- 172.21.15.78:0/3060168872 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1c1c0ab9e0 0x7f1c1c0c0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:59:06.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.060+0000 7f1c1affd700 1 --2- 172.21.15.78:0/3060168872 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c1c14b7e0 0x7f1c1c158800 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f1c1000e9b0 tx=0x7f1c1000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:06.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.060+0000 7f1c037fe700 1 -- 172.21.15.78:0/3060168872 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c10009710 con 0x7f1c1c14b7e0 2023-10-14T11:59:06.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.060+0000 7f1c037fe700 1 -- 172.21.15.78:0/3060168872 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c1000c850 con 0x7f1c1c14b7e0 2023-10-14T11:59:06.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.060+0000 7f1c037fe700 1 -- 172.21.15.78:0/3060168872 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1000cbc0 con 0x7f1c1c14b7e0 2023-10-14T11:59:06.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.060+0000 7f1c215c3700 1 -- 172.21.15.78:0/3060168872 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c1c0bf430 con 0x7f1c1c14b7e0 2023-10-14T11:59:06.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.060+0000 7f1c215c3700 1 -- 172.21.15.78:0/3060168872 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c1c0c0840 con 0x7f1c1c14b7e0 2023-10-14T11:59:06.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.061+0000 7f1c215c3700 1 -- 172.21.15.78:0/3060168872 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c1c0a9a80 con 0x7f1c1c14b7e0 2023-10-14T11:59:06.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.061+0000 7f1c037fe700 1 -- 172.21.15.78:0/3060168872 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1c1000cd20 con 0x7f1c1c14b7e0 2023-10-14T11:59:06.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.062+0000 7f1c037fe700 1 --2- 172.21.15.78:0/3060168872 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f1c0806af30 0x7f1c0806d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:06.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.062+0000 7f1c037fe700 1 -- 172.21.15.78:0/3060168872 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f1c10089660 con 0x7f1c1c14b7e0 2023-10-14T11:59:06.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.062+0000 7f1c1a7fc700 1 --2- 172.21.15.78:0/3060168872 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f1c0806af30 0x7f1c0806d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:06.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.062+0000 7f1c1a7fc700 1 --2- 172.21.15.78:0/3060168872 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f1c0806af30 0x7f1c0806d3f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c0098f0 tx=0x7f1c0c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:06.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.066+0000 7f1c037fe700 1 -- 172.21.15.78:0/3060168872 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1c100546f0 con 0x7f1c1c14b7e0 2023-10-14T11:59:06.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.256+0000 7f1c215c3700 1 -- 172.21.15.78:0/3060168872 --> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1c1c0c8a60 con 0x7f1c0806af30 2023-10-14T11:59:06.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.264+0000 7f1c037fe700 1 -- 172.21.15.78:0/3060168872 <== mgr.4104 v2:172.21.15.78:6832/3171890074 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+263555 (secure 0 0 0) 0x7f1c0c011b80 con 0x7f1c0806af30 2023-10-14T11:59:06.270 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:59:06.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.267+0000 7f1c215c3700 1 -- 172.21.15.78:0/3060168872 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f1c0806af30 msgr2=0x7f1c0806d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:06.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.267+0000 7f1c215c3700 1 --2- 172.21.15.78:0/3060168872 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f1c0806af30 0x7f1c0806d3f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c0098f0 tx=0x7f1c0c008030 comp rx=0 tx=0).stop 2023-10-14T11:59:06.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.267+0000 7f1c215c3700 1 -- 172.21.15.78:0/3060168872 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c1c14b7e0 msgr2=0x7f1c1c158800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:06.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.267+0000 7f1c215c3700 1 --2- 172.21.15.78:0/3060168872 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c1c14b7e0 0x7f1c1c158800 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f1c1000e9b0 tx=0x7f1c1000ecc0 comp rx=0 tx=0).stop 2023-10-14T11:59:06.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.267+0000 7f1c215c3700 1 -- 172.21.15.78:0/3060168872 shutdown_connections 2023-10-14T11:59:06.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.267+0000 7f1c215c3700 1 --2- 172.21.15.78:0/3060168872 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f1c0806af30 0x7f1c0806d3f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:06.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.267+0000 7f1c215c3700 1 --2- 172.21.15.78:0/3060168872 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1c1c0ab9e0 0x7f1c1c0c0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:06.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.267+0000 7f1c215c3700 1 --2- 172.21.15.78:0/3060168872 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c1c14b7e0 0x7f1c1c158800 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:06.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.267+0000 7f1c215c3700 1 --2- 172.21.15.78:0/3060168872 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1c1c154b90 0x7f1c1c157d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:06.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.267+0000 7f1c215c3700 1 -- 172.21.15.78:0/3060168872 >> 172.21.15.78:0/3060168872 conn(0x7f1c1c0ca2c0 msgr2=0x7f1c1c06b440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:06.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.267+0000 7f1c215c3700 1 -- 172.21.15.78:0/3060168872 shutdown_connections 2023-10-14T11:59:06.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.267+0000 7f1c215c3700 1 -- 172.21.15.78:0/3060168872 wait complete. 2023-10-14T11:59:06.274 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2023-10-14T11:59:06.281 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":54,"stamp":"2023-10-14T11:59:04.975403+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":73,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218932,"kb_used_data":2764,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755788,"statfs":{"total":773094113280,"available":772869926912,"internally_reserved":0,"allocated":2830336,"data_stored":2193229,"data_compressed":65244,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.817370"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.143338+0000","last_change":"2023-10-14T11:59:02.143338+0000","last_active":"2023-10-14T11:59:02.143338+0000","last_peered":"2023-10-14T11:59:02.143338+0000","last_clean":"2023-10-14T11:59:02.143338+0000","last_became_active":"2023-10-14T11:59:02.143049+0000","last_became_peered":"2023-10-14T11:59:02.143049+0000","last_unstale":"2023-10-14T11:59:02.143338+0000","last_undegraded":"2023-10-14T11:59:02.143338+0000","last_fullsized":"2023-10-14T11:59:02.143338+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:43:53.205090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":4,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.139597+0000","last_change":"2023-10-14T11:59:02.138579+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.139597+0000","last_undegraded":"2023-10-14T11:59:02.139597+0000","last_fullsized":"2023-10-14T11:59:02.139597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:35:59.785182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.150827+0000","last_change":"2023-10-14T11:59:02.144594+0000","last_active":"2023-10-14T11:59:02.150827+0000","last_peered":"2023-10-14T11:59:02.150827+0000","last_clean":"2023-10-14T11:59:02.150827+0000","last_became_active":"2023-10-14T11:59:02.144322+0000","last_became_peered":"2023-10-14T11:59:02.144322+0000","last_unstale":"2023-10-14T11:59:02.150827+0000","last_undegraded":"2023-10-14T11:59:02.150827+0000","last_fullsized":"2023-10-14T11:59:02.150827+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:08:25.866669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.146281+0000","last_change":"2023-10-14T11:59:02.146281+0000","last_active":"2023-10-14T11:59:02.146281+0000","last_peered":"2023-10-14T11:59:02.146281+0000","last_clean":"2023-10-14T11:59:02.146281+0000","last_became_active":"2023-10-14T11:59:02.146015+0000","last_became_peered":"2023-10-14T11:59:02.146015+0000","last_unstale":"2023-10-14T11:59:02.146281+0000","last_undegraded":"2023-10-14T11:59:02.146281+0000","last_fullsized":"2023-10-14T11:59:02.146281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:54:28.849250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.140518+0000","last_change":"2023-10-14T11:59:02.140518+0000","last_active":"2023-10-14T11:59:02.140518+0000","last_peered":"2023-10-14T11:59:02.140518+0000","last_clean":"2023-10-14T11:59:02.140518+0000","last_became_active":"2023-10-14T11:59:02.140238+0000","last_became_peered":"2023-10-14T11:59:02.140238+0000","last_unstale":"2023-10-14T11:59:02.140518+0000","last_undegraded":"2023-10-14T11:59:02.140518+0000","last_fullsized":"2023-10-14T11:59:02.140518+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:43:35.020618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.136431+0000","last_change":"2023-10-14T11:59:02.136431+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.136431+0000","last_undegraded":"2023-10-14T11:59:02.136431+0000","last_fullsized":"2023-10-14T11:59:02.136431+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:08:25.866669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.135963+0000","last_change":"2023-10-14T11:59:02.135963+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.135963+0000","last_undegraded":"2023-10-14T11:59:02.135963+0000","last_fullsized":"2023-10-14T11:59:02.135963+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:05:38.808781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.135986+0000","last_change":"2023-10-14T11:59:02.135986+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.135986+0000","last_undegraded":"2023-10-14T11:59:02.135986+0000","last_fullsized":"2023-10-14T11:59:02.135986+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:57:18.507681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.132676+0000","last_change":"2023-10-14T11:59:02.132676+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.132676+0000","last_undegraded":"2023-10-14T11:59:02.132676+0000","last_fullsized":"2023-10-14T11:59:02.132676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:20:02.525878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.133724+0000","last_change":"2023-10-14T11:59:02.133724+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.133724+0000","last_undegraded":"2023-10-14T11:59:02.133724+0000","last_fullsized":"2023-10-14T11:59:02.133724+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:15:00.593091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.143474+0000","last_change":"2023-10-14T11:59:02.143474+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.143474+0000","last_undegraded":"2023-10-14T11:59:02.143474+0000","last_fullsized":"2023-10-14T11:59:02.143474+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:07:15.007617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.143255+0000","last_change":"2023-10-14T11:59:02.139674+0000","last_active":"2023-10-14T11:59:02.143255+0000","last_peered":"2023-10-14T11:59:02.143255+0000","last_clean":"2023-10-14T11:59:02.143255+0000","last_became_active":"2023-10-14T11:59:02.139404+0000","last_became_peered":"2023-10-14T11:59:02.139404+0000","last_unstale":"2023-10-14T11:59:02.143255+0000","last_undegraded":"2023-10-14T11:59:02.143255+0000","last_fullsized":"2023-10-14T11:59:02.143255+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:30:41.965448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.145057+0000","last_change":"2023-10-14T11:59:02.145057+0000","last_active":"2023-10-14T11:59:02.145057+0000","last_peered":"2023-10-14T11:59:02.145057+0000","last_clean":"2023-10-14T11:59:02.145057+0000","last_became_active":"2023-10-14T11:59:02.144776+0000","last_became_peered":"2023-10-14T11:59:02.144776+0000","last_unstale":"2023-10-14T11:59:02.145057+0000","last_undegraded":"2023-10-14T11:59:02.145057+0000","last_fullsized":"2023-10-14T11:59:02.145057+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:03:39.032281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.137158+0000","last_change":"2023-10-14T11:59:02.137158+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.137158+0000","last_undegraded":"2023-10-14T11:59:02.137158+0000","last_fullsized":"2023-10-14T11:59:02.137158+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:12:09.205507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.140788+0000","last_change":"2023-10-14T11:59:02.134707+0000","last_active":"2023-10-14T11:59:02.140788+0000","last_peered":"2023-10-14T11:59:02.140788+0000","last_clean":"2023-10-14T11:59:02.140788+0000","last_became_active":"2023-10-14T11:59:02.134424+0000","last_became_peered":"2023-10-14T11:59:02.134424+0000","last_unstale":"2023-10-14T11:59:02.140788+0000","last_undegraded":"2023-10-14T11:59:02.140788+0000","last_fullsized":"2023-10-14T11:59:02.140788+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:14.681110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.138148+0000","last_change":"2023-10-14T11:59:02.138148+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.138148+0000","last_undegraded":"2023-10-14T11:59:02.138148+0000","last_fullsized":"2023-10-14T11:59:02.138148+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:27:07.788319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.136715+0000","last_change":"2023-10-14T11:59:02.136715+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.136715+0000","last_undegraded":"2023-10-14T11:59:02.136715+0000","last_fullsized":"2023-10-14T11:59:02.136715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:30:41.965448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.135337+0000","last_change":"2023-10-14T11:59:02.135337+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.135337+0000","last_undegraded":"2023-10-14T11:59:02.135337+0000","last_fullsized":"2023-10-14T11:59:02.135337+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:05:38.808781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.141545+0000","last_change":"2023-10-14T11:59:02.141545+0000","last_active":"2023-10-14T11:59:02.141545+0000","last_peered":"2023-10-14T11:59:02.141545+0000","last_clean":"2023-10-14T11:59:02.141545+0000","last_became_active":"2023-10-14T11:59:02.141281+0000","last_became_peered":"2023-10-14T11:59:02.141281+0000","last_unstale":"2023-10-14T11:59:02.141545+0000","last_undegraded":"2023-10-14T11:59:02.141545+0000","last_fullsized":"2023-10-14T11:59:02.141545+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:26.625395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.145736+0000","last_change":"2023-10-14T11:59:02.145736+0000","last_active":"2023-10-14T11:59:02.145736+0000","last_peered":"2023-10-14T11:59:02.145736+0000","last_clean":"2023-10-14T11:59:02.145736+0000","last_became_active":"2023-10-14T11:59:02.145465+0000","last_became_peered":"2023-10-14T11:59:02.145465+0000","last_unstale":"2023-10-14T11:59:02.145736+0000","last_undegraded":"2023-10-14T11:59:02.145736+0000","last_fullsized":"2023-10-14T11:59:02.145736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:08:45.915417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.132115+0000","last_change":"2023-10-14T11:59:02.132115+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.132115+0000","last_undegraded":"2023-10-14T11:59:02.132115+0000","last_fullsized":"2023-10-14T11:59:02.132115+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:53:44.574661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.132815+0000","last_change":"2023-10-14T11:59:02.132815+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.132815+0000","last_undegraded":"2023-10-14T11:59:02.132815+0000","last_fullsized":"2023-10-14T11:59:02.132815+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:39:49.443148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.137732+0000","last_change":"2023-10-14T11:59:02.137732+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.137732+0000","last_undegraded":"2023-10-14T11:59:02.137732+0000","last_fullsized":"2023-10-14T11:59:02.137732+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:54:28.849250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.150417+0000","last_change":"2023-10-14T11:59:02.149221+0000","last_active":"2023-10-14T11:59:02.150417+0000","last_peered":"2023-10-14T11:59:02.150417+0000","last_clean":"2023-10-14T11:59:02.150417+0000","last_became_active":"2023-10-14T11:59:02.148952+0000","last_became_peered":"2023-10-14T11:59:02.148952+0000","last_unstale":"2023-10-14T11:59:02.150417+0000","last_undegraded":"2023-10-14T11:59:02.150417+0000","last_fullsized":"2023-10-14T11:59:02.150417+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:58:07.847931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.146060+0000","last_change":"2023-10-14T11:59:02.146060+0000","last_active":"2023-10-14T11:59:02.146060+0000","last_peered":"2023-10-14T11:59:02.146060+0000","last_clean":"2023-10-14T11:59:02.146060+0000","last_became_active":"2023-10-14T11:59:02.145805+0000","last_became_peered":"2023-10-14T11:59:02.145805+0000","last_unstale":"2023-10-14T11:59:02.146060+0000","last_undegraded":"2023-10-14T11:59:02.146060+0000","last_fullsized":"2023-10-14T11:59:02.146060+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:39:49.443148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.132003+0000","last_change":"2023-10-14T11:59:02.132003+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.132003+0000","last_undegraded":"2023-10-14T11:59:02.132003+0000","last_fullsized":"2023-10-14T11:59:02.132003+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:54:28.849250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.146857+0000","last_change":"2023-10-14T11:59:02.146857+0000","last_active":"2023-10-14T11:59:02.146857+0000","last_peered":"2023-10-14T11:59:02.146857+0000","last_clean":"2023-10-14T11:59:02.146857+0000","last_became_active":"2023-10-14T11:59:02.146627+0000","last_became_peered":"2023-10-14T11:59:02.146627+0000","last_unstale":"2023-10-14T11:59:02.146857+0000","last_undegraded":"2023-10-14T11:59:02.146857+0000","last_fullsized":"2023-10-14T11:59:02.146857+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:30:41.965448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.149774+0000","last_change":"2023-10-14T11:59:02.143747+0000","last_active":"2023-10-14T11:59:02.149774+0000","last_peered":"2023-10-14T11:59:02.149774+0000","last_clean":"2023-10-14T11:59:02.149774+0000","last_became_active":"2023-10-14T11:59:02.143421+0000","last_became_peered":"2023-10-14T11:59:02.143421+0000","last_unstale":"2023-10-14T11:59:02.149774+0000","last_undegraded":"2023-10-14T11:59:02.149774+0000","last_fullsized":"2023-10-14T11:59:02.149774+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:27:07.788319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.139034+0000","last_change":"2023-10-14T11:59:02.139034+0000","last_active":"2023-10-14T11:59:02.139034+0000","last_peered":"2023-10-14T11:59:02.139034+0000","last_clean":"2023-10-14T11:59:02.139034+0000","last_became_active":"2023-10-14T11:59:02.138730+0000","last_became_peered":"2023-10-14T11:59:02.138730+0000","last_unstale":"2023-10-14T11:59:02.139034+0000","last_undegraded":"2023-10-14T11:59:02.139034+0000","last_fullsized":"2023-10-14T11:59:02.139034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:48:37.789205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.136790+0000","last_change":"2023-10-14T11:59:02.136790+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.136790+0000","last_undegraded":"2023-10-14T11:59:02.136790+0000","last_fullsized":"2023-10-14T11:59:02.136790+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:58:07.847931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.131455+0000","last_change":"2023-10-14T11:59:02.131455+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.131455+0000","last_undegraded":"2023-10-14T11:59:02.131455+0000","last_fullsized":"2023-10-14T11:59:02.131455+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:05:58.896548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.138555+0000","last_change":"2023-10-14T11:59:02.138555+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.138555+0000","last_undegraded":"2023-10-14T11:59:02.138555+0000","last_fullsized":"2023-10-14T11:59:02.138555+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:08:25.866669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.136873+0000","last_change":"2023-10-14T11:59:02.136873+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.136873+0000","last_undegraded":"2023-10-14T11:59:02.136873+0000","last_fullsized":"2023-10-14T11:59:02.136873+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:20:02.525878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.133827+0000","last_change":"2023-10-14T11:59:01.170225+0000","last_active":"2023-10-14T11:59:02.133827+0000","last_peered":"2023-10-14T11:59:02.133827+0000","last_clean":"2023-10-14T11:59:02.133827+0000","last_became_active":"2023-10-14T11:59:01.169927+0000","last_became_peered":"2023-10-14T11:59:01.169927+0000","last_unstale":"2023-10-14T11:59:02.133827+0000","last_undegraded":"2023-10-14T11:59:02.133827+0000","last_fullsized":"2023-10-14T11:59:02.133827+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:25.623744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.139277+0000","last_change":"2023-10-14T11:59:02.139277+0000","last_active":"2023-10-14T11:59:02.139277+0000","last_peered":"2023-10-14T11:59:02.139277+0000","last_clean":"2023-10-14T11:59:02.139277+0000","last_became_active":"2023-10-14T11:59:02.138985+0000","last_became_peered":"2023-10-14T11:59:02.138985+0000","last_unstale":"2023-10-14T11:59:02.139277+0000","last_undegraded":"2023-10-14T11:59:02.139277+0000","last_fullsized":"2023-10-14T11:59:02.139277+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:15:00.593091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.137138+0000","last_change":"2023-10-14T11:59:01.178406+0000","last_active":"2023-10-14T11:59:02.137138+0000","last_peered":"2023-10-14T11:59:02.137138+0000","last_clean":"2023-10-14T11:59:02.137138+0000","last_became_active":"2023-10-14T11:59:01.178136+0000","last_became_peered":"2023-10-14T11:59:01.178136+0000","last_unstale":"2023-10-14T11:59:02.137138+0000","last_undegraded":"2023-10-14T11:59:02.137138+0000","last_fullsized":"2023-10-14T11:59:02.137138+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:43:52.203439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.139738+0000","last_change":"2023-10-14T11:59:02.139738+0000","last_active":"2023-10-14T11:59:02.139738+0000","last_peered":"2023-10-14T11:59:02.139738+0000","last_clean":"2023-10-14T11:59:02.139738+0000","last_became_active":"2023-10-14T11:59:02.139431+0000","last_became_peered":"2023-10-14T11:59:02.139431+0000","last_unstale":"2023-10-14T11:59:02.139738+0000","last_undegraded":"2023-10-14T11:59:02.139738+0000","last_fullsized":"2023-10-14T11:59:02.139738+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:41:47.404184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.133347+0000","last_change":"2023-10-14T11:59:01.179865+0000","last_active":"2023-10-14T11:59:02.133347+0000","last_peered":"2023-10-14T11:59:02.133347+0000","last_clean":"2023-10-14T11:59:02.133347+0000","last_became_active":"2023-10-14T11:59:01.178404+0000","last_became_peered":"2023-10-14T11:59:01.178404+0000","last_unstale":"2023-10-14T11:59:02.133347+0000","last_undegraded":"2023-10-14T11:59:02.133347+0000","last_fullsized":"2023-10-14T11:59:02.133347+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:30:40.963797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.146195+0000","last_change":"2023-10-14T11:59:02.146195+0000","last_active":"2023-10-14T11:59:02.146195+0000","last_peered":"2023-10-14T11:59:02.146195+0000","last_clean":"2023-10-14T11:59:02.146195+0000","last_became_active":"2023-10-14T11:59:02.144184+0000","last_became_peered":"2023-10-14T11:59:02.144184+0000","last_unstale":"2023-10-14T11:59:02.146195+0000","last_undegraded":"2023-10-14T11:59:02.146195+0000","last_fullsized":"2023-10-14T11:59:02.146195+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:43:53.205090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.135778+0000","last_change":"2023-10-14T11:59:01.185486+0000","last_active":"2023-10-14T11:59:02.135778+0000","last_peered":"2023-10-14T11:59:02.135778+0000","last_clean":"2023-10-14T11:59:02.135778+0000","last_became_active":"2023-10-14T11:59:01.185200+0000","last_became_peered":"2023-10-14T11:59:01.185200+0000","last_unstale":"2023-10-14T11:59:02.135778+0000","last_undegraded":"2023-10-14T11:59:02.135778+0000","last_fullsized":"2023-10-14T11:59:02.135778+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:48:36.787554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.139052+0000","last_change":"2023-10-14T11:59:02.139052+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.139052+0000","last_undegraded":"2023-10-14T11:59:02.139052+0000","last_fullsized":"2023-10-14T11:59:02.139052+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:36:52.548523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.132966+0000","last_change":"2023-10-14T11:59:01.178774+0000","last_active":"2023-10-14T11:59:02.132966+0000","last_peered":"2023-10-14T11:59:02.132966+0000","last_clean":"2023-10-14T11:59:02.132966+0000","last_became_active":"2023-10-14T11:59:01.178378+0000","last_became_peered":"2023-10-14T11:59:01.178378+0000","last_unstale":"2023-10-14T11:59:02.132966+0000","last_undegraded":"2023-10-14T11:59:02.132966+0000","last_fullsized":"2023-10-14T11:59:02.132966+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:25.623744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.149841+0000","last_change":"2023-10-14T11:59:02.143614+0000","last_active":"2023-10-14T11:59:02.149841+0000","last_peered":"2023-10-14T11:59:02.149841+0000","last_clean":"2023-10-14T11:59:02.149841+0000","last_became_active":"2023-10-14T11:59:02.143332+0000","last_became_peered":"2023-10-14T11:59:02.143332+0000","last_unstale":"2023-10-14T11:59:02.149841+0000","last_undegraded":"2023-10-14T11:59:02.149841+0000","last_fullsized":"2023-10-14T11:59:02.149841+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:12:09.205507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.133277+0000","last_change":"2023-10-14T11:59:01.178778+0000","last_active":"2023-10-14T11:59:02.133277+0000","last_peered":"2023-10-14T11:59:02.133277+0000","last_clean":"2023-10-14T11:59:02.133277+0000","last_became_active":"2023-10-14T11:59:01.178440+0000","last_became_peered":"2023-10-14T11:59:01.178440+0000","last_unstale":"2023-10-14T11:59:02.133277+0000","last_undegraded":"2023-10-14T11:59:02.133277+0000","last_fullsized":"2023-10-14T11:59:02.133277+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:18:16.224262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.139185+0000","last_change":"2023-10-14T11:59:02.139185+0000","last_active":"2023-10-14T11:59:02.139185+0000","last_peered":"2023-10-14T11:59:02.139185+0000","last_clean":"2023-10-14T11:59:02.139185+0000","last_became_active":"2023-10-14T11:59:02.138862+0000","last_became_peered":"2023-10-14T11:59:02.138862+0000","last_unstale":"2023-10-14T11:59:02.139185+0000","last_undegraded":"2023-10-14T11:59:02.139185+0000","last_fullsized":"2023-10-14T11:59:02.139185+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:48:45.563805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.132488+0000","last_change":"2023-10-14T11:59:01.175662+0000","last_active":"2023-10-14T11:59:02.132488+0000","last_peered":"2023-10-14T11:59:02.132488+0000","last_clean":"2023-10-14T11:59:02.132488+0000","last_became_active":"2023-10-14T11:59:01.175405+0000","last_became_peered":"2023-10-14T11:59:01.175405+0000","last_unstale":"2023-10-14T11:59:02.132488+0000","last_undegraded":"2023-10-14T11:59:02.132488+0000","last_fullsized":"2023-10-14T11:59:02.132488+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:57:17.506030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.143559+0000","last_change":"2023-10-14T11:59:02.143559+0000","last_active":"2023-10-14T11:59:02.143559+0000","last_peered":"2023-10-14T11:59:02.143559+0000","last_clean":"2023-10-14T11:59:02.143559+0000","last_became_active":"2023-10-14T11:59:02.143303+0000","last_became_peered":"2023-10-14T11:59:02.143303+0000","last_unstale":"2023-10-14T11:59:02.143559+0000","last_undegraded":"2023-10-14T11:59:02.143559+0000","last_fullsized":"2023-10-14T11:59:02.143559+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:20:02.525878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.132854+0000","last_change":"2023-10-14T11:59:01.167233+0000","last_active":"2023-10-14T11:59:02.132854+0000","last_peered":"2023-10-14T11:59:02.132854+0000","last_clean":"2023-10-14T11:59:02.132854+0000","last_became_active":"2023-10-14T11:59:01.166905+0000","last_became_peered":"2023-10-14T11:59:01.166905+0000","last_unstale":"2023-10-14T11:59:02.132854+0000","last_undegraded":"2023-10-14T11:59:02.132854+0000","last_fullsized":"2023-10-14T11:59:02.132854+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:43:52.203439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.152247+0000","last_change":"2023-10-14T11:59:02.150665+0000","last_active":"2023-10-14T11:59:02.152247+0000","last_peered":"2023-10-14T11:59:02.152247+0000","last_clean":"2023-10-14T11:59:02.152247+0000","last_became_active":"2023-10-14T11:59:02.150384+0000","last_became_peered":"2023-10-14T11:59:02.150384+0000","last_unstale":"2023-10-14T11:59:02.152247+0000","last_undegraded":"2023-10-14T11:59:02.152247+0000","last_fullsized":"2023-10-14T11:59:02.152247+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:07:15.007617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.133851+0000","last_change":"2023-10-14T11:59:01.179314+0000","last_active":"2023-10-14T11:59:02.133851+0000","last_peered":"2023-10-14T11:59:02.133851+0000","last_clean":"2023-10-14T11:59:02.133851+0000","last_became_active":"2023-10-14T11:59:01.179019+0000","last_became_peered":"2023-10-14T11:59:01.179019+0000","last_unstale":"2023-10-14T11:59:02.133851+0000","last_undegraded":"2023-10-14T11:59:02.133851+0000","last_fullsized":"2023-10-14T11:59:02.133851+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:27:06.786668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.137973+0000","last_change":"2023-10-14T11:59:02.137973+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.137973+0000","last_undegraded":"2023-10-14T11:59:02.137973+0000","last_fullsized":"2023-10-14T11:59:02.137973+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:36:52.548523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.137264+0000","last_change":"2023-10-14T11:59:01.177873+0000","last_active":"2023-10-14T11:59:02.137264+0000","last_peered":"2023-10-14T11:59:02.137264+0000","last_clean":"2023-10-14T11:59:02.137264+0000","last_became_active":"2023-10-14T11:59:01.177594+0000","last_became_peered":"2023-10-14T11:59:01.177594+0000","last_unstale":"2023-10-14T11:59:02.137264+0000","last_undegraded":"2023-10-14T11:59:02.137264+0000","last_fullsized":"2023-10-14T11:59:02.137264+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:25.623744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.132187+0000","last_change":"2023-10-14T11:59:02.132187+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.132187+0000","last_undegraded":"2023-10-14T11:59:02.132187+0000","last_fullsized":"2023-10-14T11:59:02.132187+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:47:13.361893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.135909+0000","last_change":"2023-10-14T11:59:01.178143+0000","last_active":"2023-10-14T11:59:02.135909+0000","last_peered":"2023-10-14T11:59:02.135909+0000","last_clean":"2023-10-14T11:59:02.135909+0000","last_became_active":"2023-10-14T11:59:01.177736+0000","last_became_peered":"2023-10-14T11:59:01.177736+0000","last_unstale":"2023-10-14T11:59:02.135909+0000","last_undegraded":"2023-10-14T11:59:02.135909+0000","last_fullsized":"2023-10-14T11:59:02.135909+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:35:58.783531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.150847+0000","last_change":"2023-10-14T11:59:02.145816+0000","last_active":"2023-10-14T11:59:02.150847+0000","last_peered":"2023-10-14T11:59:02.150847+0000","last_clean":"2023-10-14T11:59:02.150847+0000","last_became_active":"2023-10-14T11:59:02.145548+0000","last_became_peered":"2023-10-14T11:59:02.145548+0000","last_unstale":"2023-10-14T11:59:02.150847+0000","last_undegraded":"2023-10-14T11:59:02.150847+0000","last_fullsized":"2023-10-14T11:59:02.150847+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:13:03.743718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.133733+0000","last_change":"2023-10-14T11:59:01.180187+0000","last_active":"2023-10-14T11:59:02.133733+0000","last_peered":"2023-10-14T11:59:02.133733+0000","last_clean":"2023-10-14T11:59:02.133733+0000","last_became_active":"2023-10-14T11:59:01.179909+0000","last_became_peered":"2023-10-14T11:59:01.179909+0000","last_unstale":"2023-10-14T11:59:02.133733+0000","last_undegraded":"2023-10-14T11:59:02.133733+0000","last_fullsized":"2023-10-14T11:59:02.133733+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:58:06.846280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.143696+0000","last_change":"2023-10-14T11:59:02.143696+0000","last_active":"2023-10-14T11:59:02.143696+0000","last_peered":"2023-10-14T11:59:02.143696+0000","last_clean":"2023-10-14T11:59:02.143696+0000","last_became_active":"2023-10-14T11:59:02.143443+0000","last_became_peered":"2023-10-14T11:59:02.143443+0000","last_unstale":"2023-10-14T11:59:02.143696+0000","last_undegraded":"2023-10-14T11:59:02.143696+0000","last_fullsized":"2023-10-14T11:59:02.143696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:47:13.361893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.134332+0000","last_change":"2023-10-14T11:59:01.165901+0000","last_active":"2023-10-14T11:59:02.134332+0000","last_peered":"2023-10-14T11:59:02.134332+0000","last_clean":"2023-10-14T11:59:02.134332+0000","last_became_active":"2023-10-14T11:59:01.165597+0000","last_became_peered":"2023-10-14T11:59:01.165597+0000","last_unstale":"2023-10-14T11:59:02.134332+0000","last_undegraded":"2023-10-14T11:59:02.134332+0000","last_fullsized":"2023-10-14T11:59:02.134332+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:05:37.807130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.141427+0000","last_change":"2023-10-14T11:59:02.141427+0000","last_active":"2023-10-14T11:59:02.141427+0000","last_peered":"2023-10-14T11:59:02.141427+0000","last_clean":"2023-10-14T11:59:02.141427+0000","last_became_active":"2023-10-14T11:59:02.141166+0000","last_became_peered":"2023-10-14T11:59:02.141166+0000","last_unstale":"2023-10-14T11:59:02.141427+0000","last_undegraded":"2023-10-14T11:59:02.141427+0000","last_fullsized":"2023-10-14T11:59:02.141427+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:18:17.225913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.136387+0000","last_change":"2023-10-14T11:59:01.172037+0000","last_active":"2023-10-14T11:59:02.136387+0000","last_peered":"2023-10-14T11:59:02.136387+0000","last_clean":"2023-10-14T11:59:02.136387+0000","last_became_active":"2023-10-14T11:59:01.170760+0000","last_became_peered":"2023-10-14T11:59:01.170760+0000","last_unstale":"2023-10-14T11:59:02.136387+0000","last_undegraded":"2023-10-14T11:59:02.136387+0000","last_fullsized":"2023-10-14T11:59:02.136387+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:57:17.506030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.138925+0000","last_change":"2023-10-14T11:59:02.138925+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.138925+0000","last_undegraded":"2023-10-14T11:59:02.138925+0000","last_fullsized":"2023-10-14T11:59:02.138925+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:12:09.205507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.132642+0000","last_change":"2023-10-14T11:59:01.178960+0000","last_active":"2023-10-14T11:59:02.132642+0000","last_peered":"2023-10-14T11:59:02.132642+0000","last_clean":"2023-10-14T11:59:02.132642+0000","last_became_active":"2023-10-14T11:59:01.178691+0000","last_became_peered":"2023-10-14T11:59:01.178691+0000","last_unstale":"2023-10-14T11:59:02.132642+0000","last_undegraded":"2023-10-14T11:59:02.132642+0000","last_fullsized":"2023-10-14T11:59:02.132642+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:13.679459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.140181+0000","last_change":"2023-10-14T11:59:02.140181+0000","last_active":"2023-10-14T11:59:02.140181+0000","last_peered":"2023-10-14T11:59:02.140181+0000","last_clean":"2023-10-14T11:59:02.140181+0000","last_became_active":"2023-10-14T11:59:02.139382+0000","last_became_peered":"2023-10-14T11:59:02.139382+0000","last_unstale":"2023-10-14T11:59:02.140181+0000","last_undegraded":"2023-10-14T11:59:02.140181+0000","last_fullsized":"2023-10-14T11:59:02.140181+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:20:02.525878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.138336+0000","last_change":"2023-10-14T11:59:01.178769+0000","last_active":"2023-10-14T11:59:02.138336+0000","last_peered":"2023-10-14T11:59:02.138336+0000","last_clean":"2023-10-14T11:59:02.138336+0000","last_became_active":"2023-10-14T11:59:01.178288+0000","last_became_peered":"2023-10-14T11:59:01.178288+0000","last_unstale":"2023-10-14T11:59:02.138336+0000","last_undegraded":"2023-10-14T11:59:02.138336+0000","last_fullsized":"2023-10-14T11:59:02.138336+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:07:14.005966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.132368+0000","last_change":"2023-10-14T11:59:02.132368+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.132368+0000","last_undegraded":"2023-10-14T11:59:02.132368+0000","last_fullsized":"2023-10-14T11:59:02.132368+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:05:58.896548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.134614+0000","last_change":"2023-10-14T11:59:01.183630+0000","last_active":"2023-10-14T11:59:02.134614+0000","last_peered":"2023-10-14T11:59:02.134614+0000","last_clean":"2023-10-14T11:59:02.134614+0000","last_became_active":"2023-10-14T11:59:01.183408+0000","last_became_peered":"2023-10-14T11:59:01.183408+0000","last_unstale":"2023-10-14T11:59:02.134614+0000","last_undegraded":"2023-10-14T11:59:02.134614+0000","last_fullsized":"2023-10-14T11:59:02.134614+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:25.623744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.145545+0000","last_change":"2023-10-14T11:59:02.141878+0000","last_active":"2023-10-14T11:59:02.145545+0000","last_peered":"2023-10-14T11:59:02.145545+0000","last_clean":"2023-10-14T11:59:02.145545+0000","last_became_active":"2023-10-14T11:59:02.141229+0000","last_became_peered":"2023-10-14T11:59:02.141229+0000","last_unstale":"2023-10-14T11:59:02.145545+0000","last_undegraded":"2023-10-14T11:59:02.145545+0000","last_fullsized":"2023-10-14T11:59:02.145545+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:11:18.803472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.132715+0000","last_change":"2023-10-14T11:59:01.165942+0000","last_active":"2023-10-14T11:59:02.132715+0000","last_peered":"2023-10-14T11:59:02.132715+0000","last_clean":"2023-10-14T11:59:02.132715+0000","last_became_active":"2023-10-14T11:59:01.165646+0000","last_became_peered":"2023-10-14T11:59:01.165646+0000","last_unstale":"2023-10-14T11:59:02.132715+0000","last_undegraded":"2023-10-14T11:59:02.132715+0000","last_fullsized":"2023-10-14T11:59:02.132715+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:14:59.591440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.147269+0000","last_change":"2023-10-14T11:59:02.147269+0000","last_active":"2023-10-14T11:59:02.147269+0000","last_peered":"2023-10-14T11:59:02.147269+0000","last_clean":"2023-10-14T11:59:02.147269+0000","last_became_active":"2023-10-14T11:59:02.147005+0000","last_became_peered":"2023-10-14T11:59:02.147005+0000","last_unstale":"2023-10-14T11:59:02.147269+0000","last_undegraded":"2023-10-14T11:59:02.147269+0000","last_fullsized":"2023-10-14T11:59:02.147269+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:18:17.225913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.131810+0000","last_change":"2023-10-14T11:59:01.183080+0000","last_active":"2023-10-14T11:59:02.131810+0000","last_peered":"2023-10-14T11:59:02.131810+0000","last_clean":"2023-10-14T11:59:02.131810+0000","last_became_active":"2023-10-14T11:59:01.182786+0000","last_became_peered":"2023-10-14T11:59:01.182786+0000","last_unstale":"2023-10-14T11:59:02.131810+0000","last_undegraded":"2023-10-14T11:59:02.131810+0000","last_fullsized":"2023-10-14T11:59:02.131810+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:05:27.508197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.146618+0000","last_change":"2023-10-14T11:59:02.146618+0000","last_active":"2023-10-14T11:59:02.146618+0000","last_peered":"2023-10-14T11:59:02.146618+0000","last_clean":"2023-10-14T11:59:02.146618+0000","last_became_active":"2023-10-14T11:59:02.146352+0000","last_became_peered":"2023-10-14T11:59:02.146352+0000","last_unstale":"2023-10-14T11:59:02.146618+0000","last_undegraded":"2023-10-14T11:59:02.146618+0000","last_fullsized":"2023-10-14T11:59:02.146618+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:48:45.563805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.132063+0000","last_change":"2023-10-14T11:59:01.178771+0000","last_active":"2023-10-14T11:59:02.132063+0000","last_peered":"2023-10-14T11:59:02.132063+0000","last_clean":"2023-10-14T11:59:02.132063+0000","last_became_active":"2023-10-14T11:59:01.178416+0000","last_became_peered":"2023-10-14T11:59:01.178416+0000","last_unstale":"2023-10-14T11:59:02.132063+0000","last_undegraded":"2023-10-14T11:59:02.132063+0000","last_fullsized":"2023-10-14T11:59:02.132063+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:11:46.659890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.145624+0000","last_change":"2023-10-14T11:59:02.145624+0000","last_active":"2023-10-14T11:59:02.145624+0000","last_peered":"2023-10-14T11:59:02.145624+0000","last_clean":"2023-10-14T11:59:02.145624+0000","last_became_active":"2023-10-14T11:59:02.145381+0000","last_became_peered":"2023-10-14T11:59:02.145381+0000","last_unstale":"2023-10-14T11:59:02.145624+0000","last_undegraded":"2023-10-14T11:59:02.145624+0000","last_fullsized":"2023-10-14T11:59:02.145624+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:21:57.196717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.137214+0000","last_change":"2023-10-14T11:59:01.177556+0000","last_active":"2023-10-14T11:59:02.137214+0000","last_peered":"2023-10-14T11:59:02.137214+0000","last_clean":"2023-10-14T11:59:02.137214+0000","last_became_active":"2023-10-14T11:59:01.177313+0000","last_became_peered":"2023-10-14T11:59:01.177313+0000","last_unstale":"2023-10-14T11:59:02.137214+0000","last_undegraded":"2023-10-14T11:59:02.137214+0000","last_fullsized":"2023-10-14T11:59:02.137214+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:27:02.969220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.141409+0000","last_change":"2023-10-14T11:59:02.141409+0000","last_active":"2023-10-14T11:59:02.141409+0000","last_peered":"2023-10-14T11:59:02.141409+0000","last_clean":"2023-10-14T11:59:02.141409+0000","last_became_active":"2023-10-14T11:59:02.141163+0000","last_became_peered":"2023-10-14T11:59:02.141163+0000","last_unstale":"2023-10-14T11:59:02.141409+0000","last_undegraded":"2023-10-14T11:59:02.141409+0000","last_fullsized":"2023-10-14T11:59:02.141409+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:54:06.957363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.133688+0000","last_change":"2023-10-14T11:59:01.168240+0000","last_active":"2023-10-14T11:59:02.133688+0000","last_peered":"2023-10-14T11:59:02.133688+0000","last_clean":"2023-10-14T11:59:02.133688+0000","last_became_active":"2023-10-14T11:59:01.167944+0000","last_became_peered":"2023-10-14T11:59:01.167944+0000","last_unstale":"2023-10-14T11:59:02.133688+0000","last_undegraded":"2023-10-14T11:59:02.133688+0000","last_fullsized":"2023-10-14T11:59:02.133688+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:27:02.969220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":4,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.140710+0000","last_change":"2023-10-14T11:59:02.133062+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.140710+0000","last_undegraded":"2023-10-14T11:59:02.140710+0000","last_fullsized":"2023-10-14T11:59:02.140710+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:40:14.734059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.133269+0000","last_change":"2023-10-14T11:59:01.176530+0000","last_active":"2023-10-14T11:59:02.133269+0000","last_peered":"2023-10-14T11:59:02.133269+0000","last_clean":"2023-10-14T11:59:02.133269+0000","last_became_active":"2023-10-14T11:59:01.176193+0000","last_became_peered":"2023-10-14T11:59:01.176193+0000","last_unstale":"2023-10-14T11:59:02.133269+0000","last_undegraded":"2023-10-14T11:59:02.133269+0000","last_fullsized":"2023-10-14T11:59:02.133269+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:54:27.847599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.136372+0000","last_change":"2023-10-14T11:59:02.136372+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.136372+0000","last_undegraded":"2023-10-14T11:59:02.136372+0000","last_fullsized":"2023-10-14T11:59:02.136372+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:14.681110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.130535+0000","last_change":"2023-10-14T11:59:01.178157+0000","last_active":"2023-10-14T11:59:02.130535+0000","last_peered":"2023-10-14T11:59:02.130535+0000","last_clean":"2023-10-14T11:59:02.130535+0000","last_became_active":"2023-10-14T11:59:01.177863+0000","last_became_peered":"2023-10-14T11:59:01.177863+0000","last_unstale":"2023-10-14T11:59:02.130535+0000","last_undegraded":"2023-10-14T11:59:02.130535+0000","last_fullsized":"2023-10-14T11:59:02.130535+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:57:17.506030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.143662+0000","last_change":"2023-10-14T11:59:02.143662+0000","last_active":"2023-10-14T11:59:02.143662+0000","last_peered":"2023-10-14T11:59:02.143662+0000","last_clean":"2023-10-14T11:59:02.143662+0000","last_became_active":"2023-10-14T11:59:02.143456+0000","last_became_peered":"2023-10-14T11:59:02.143456+0000","last_unstale":"2023-10-14T11:59:02.143662+0000","last_undegraded":"2023-10-14T11:59:02.143662+0000","last_fullsized":"2023-10-14T11:59:02.143662+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:33:27.410202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.134465+0000","last_change":"2023-10-14T11:59:01.181546+0000","last_active":"2023-10-14T11:59:02.134465+0000","last_peered":"2023-10-14T11:59:02.134465+0000","last_clean":"2023-10-14T11:59:02.134465+0000","last_became_active":"2023-10-14T11:59:01.181255+0000","last_became_peered":"2023-10-14T11:59:01.181255+0000","last_unstale":"2023-10-14T11:59:02.134465+0000","last_undegraded":"2023-10-14T11:59:02.134465+0000","last_fullsized":"2023-10-14T11:59:02.134465+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:27:02.969220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.145565+0000","last_change":"2023-10-14T11:59:02.142119+0000","last_active":"2023-10-14T11:59:02.145565+0000","last_peered":"2023-10-14T11:59:02.145565+0000","last_clean":"2023-10-14T11:59:02.145565+0000","last_became_active":"2023-10-14T11:59:02.141586+0000","last_became_peered":"2023-10-14T11:59:02.141586+0000","last_unstale":"2023-10-14T11:59:02.145565+0000","last_undegraded":"2023-10-14T11:59:02.145565+0000","last_fullsized":"2023-10-14T11:59:02.145565+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:36:52.548523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.137387+0000","last_change":"2023-10-14T11:59:01.180897+0000","last_active":"2023-10-14T11:59:02.137387+0000","last_peered":"2023-10-14T11:59:02.137387+0000","last_clean":"2023-10-14T11:59:02.137387+0000","last_became_active":"2023-10-14T11:59:01.180524+0000","last_became_peered":"2023-10-14T11:59:01.180524+0000","last_unstale":"2023-10-14T11:59:02.137387+0000","last_undegraded":"2023-10-14T11:59:02.137387+0000","last_fullsized":"2023-10-14T11:59:02.137387+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:18:16.224262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.136894+0000","last_change":"2023-10-14T11:59:02.136894+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.136894+0000","last_undegraded":"2023-10-14T11:59:02.136894+0000","last_fullsized":"2023-10-14T11:59:02.136894+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:15:00.593091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.136362+0000","last_change":"2023-10-14T11:59:01.183591+0000","last_active":"2023-10-14T11:59:02.136362+0000","last_peered":"2023-10-14T11:59:02.136362+0000","last_clean":"2023-10-14T11:59:02.136362+0000","last_became_active":"2023-10-14T11:59:01.183019+0000","last_became_peered":"2023-10-14T11:59:01.183019+0000","last_unstale":"2023-10-14T11:59:02.136362+0000","last_undegraded":"2023-10-14T11:59:02.136362+0000","last_fullsized":"2023-10-14T11:59:02.136362+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:05:37.807130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.133356+0000","last_change":"2023-10-14T11:59:02.133356+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.133356+0000","last_undegraded":"2023-10-14T11:59:02.133356+0000","last_fullsized":"2023-10-14T11:59:02.133356+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:39:49.443148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.137080+0000","last_change":"2023-10-14T11:59:01.177539+0000","last_active":"2023-10-14T11:59:02.137080+0000","last_peered":"2023-10-14T11:59:02.137080+0000","last_clean":"2023-10-14T11:59:02.137080+0000","last_became_active":"2023-10-14T11:59:01.177255+0000","last_became_peered":"2023-10-14T11:59:01.177255+0000","last_unstale":"2023-10-14T11:59:02.137080+0000","last_undegraded":"2023-10-14T11:59:02.137080+0000","last_fullsized":"2023-10-14T11:59:02.137080+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:39:48.441497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.137750+0000","last_change":"2023-10-14T11:59:02.137750+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.137750+0000","last_undegraded":"2023-10-14T11:59:02.137750+0000","last_fullsized":"2023-10-14T11:59:02.137750+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:47:13.361893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.136562+0000","last_change":"2023-10-14T11:59:01.182837+0000","last_active":"2023-10-14T11:59:02.136562+0000","last_peered":"2023-10-14T11:59:02.136562+0000","last_clean":"2023-10-14T11:59:02.136562+0000","last_became_active":"2023-10-14T11:59:01.182236+0000","last_became_peered":"2023-10-14T11:59:01.182236+0000","last_unstale":"2023-10-14T11:59:02.136562+0000","last_undegraded":"2023-10-14T11:59:02.136562+0000","last_fullsized":"2023-10-14T11:59:02.136562+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:13.679459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":4,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.138516+0000","last_change":"2023-10-14T11:59:02.138103+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.138516+0000","last_undegraded":"2023-10-14T11:59:02.138516+0000","last_fullsized":"2023-10-14T11:59:02.138516+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:13:03.743718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.133395+0000","last_change":"2023-10-14T11:59:01.168077+0000","last_active":"2023-10-14T11:59:02.133395+0000","last_peered":"2023-10-14T11:59:02.133395+0000","last_clean":"2023-10-14T11:59:02.133395+0000","last_became_active":"2023-10-14T11:59:01.167803+0000","last_became_peered":"2023-10-14T11:59:01.167803+0000","last_unstale":"2023-10-14T11:59:02.133395+0000","last_undegraded":"2023-10-14T11:59:02.133395+0000","last_fullsized":"2023-10-14T11:59:02.133395+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:47:12.360242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.139387+0000","last_change":"2023-10-14T11:59:02.139387+0000","last_active":"2023-10-14T11:59:02.139387+0000","last_peered":"2023-10-14T11:59:02.139387+0000","last_clean":"2023-10-14T11:59:02.139387+0000","last_became_active":"2023-10-14T11:59:02.139117+0000","last_became_peered":"2023-10-14T11:59:02.139117+0000","last_unstale":"2023-10-14T11:59:02.139387+0000","last_undegraded":"2023-10-14T11:59:02.139387+0000","last_fullsized":"2023-10-14T11:59:02.139387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:27:03.970871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.138469+0000","last_change":"2023-10-14T11:59:01.179436+0000","last_active":"2023-10-14T11:59:02.138469+0000","last_peered":"2023-10-14T11:59:02.138469+0000","last_clean":"2023-10-14T11:59:02.138469+0000","last_became_active":"2023-10-14T11:59:01.179109+0000","last_became_peered":"2023-10-14T11:59:01.179109+0000","last_unstale":"2023-10-14T11:59:02.138469+0000","last_undegraded":"2023-10-14T11:59:02.138469+0000","last_fullsized":"2023-10-14T11:59:02.138469+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:11:17.801821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.143298+0000","last_change":"2023-10-14T11:59:02.143298+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.143298+0000","last_undegraded":"2023-10-14T11:59:02.143298+0000","last_fullsized":"2023-10-14T11:59:02.143298+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:13:03.743718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.131329+0000","last_change":"2023-10-14T11:59:01.169175+0000","last_active":"2023-10-14T11:59:02.131329+0000","last_peered":"2023-10-14T11:59:02.131329+0000","last_clean":"2023-10-14T11:59:02.131329+0000","last_became_active":"2023-10-14T11:59:01.168873+0000","last_became_peered":"2023-10-14T11:59:01.168873+0000","last_unstale":"2023-10-14T11:59:02.131329+0000","last_undegraded":"2023-10-14T11:59:02.131329+0000","last_fullsized":"2023-10-14T11:59:02.131329+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:53:43.573010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":172,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-14T11:59:01.174640+0000","last_change":"2023-10-14T11:58:10.616726+0000","last_active":"2023-10-14T11:59:01.174640+0000","last_peered":"2023-10-14T11:59:01.174640+0000","last_clean":"2023-10-14T11:59:01.174640+0000","last_became_active":"2023-10-14T11:58:10.615628+0000","last_became_peered":"2023-10-14T11:58:10.615628+0000","last_unstale":"2023-10-14T11:59:01.174640+0000","last_undegraded":"2023-10-14T11:59:01.174640+0000","last_fullsized":"2023-10-14T11:59:01.174640+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:09.001265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:09.001265+0000","last_clean_scrub_stamp":"2023-10-14T11:58:09.001265+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:22:50.496050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65244,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574862,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":716079,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574862,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":125711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84399999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574862,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":132516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]}]},{"osd":0,"up_from":13,"seq":55834574861,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":401408,"data_stored":716079,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574861,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":125711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.254}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574862,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":125711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574862,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":125711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574862,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":125711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.311}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T11:59:06.287 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T11:59:06.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.441+0000 7f83092b8700 1 Processor -- start 2023-10-14T11:59:06.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.441+0000 7f83092b8700 1 -- start start 2023-10-14T11:59:06.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.441+0000 7f83092b8700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f830406d6b0 0x7f83040c6270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:06.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.441+0000 7f83092b8700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f83040c7340 0x7f83040c67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:06.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.441+0000 7f83092b8700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8304158450 con 0x7f8304152110 2023-10-14T11:59:06.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.441+0000 7f83092b8700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83041590d0 con 0x7f83040c7340 2023-10-14T11:59:06.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.441+0000 7f83092b8700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8304158730 con 0x7f830406d6b0 2023-10-14T11:59:06.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.441+0000 7f83037fe700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f83040c7340 0x7f83040c67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:06.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.441+0000 7f8302ffd700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f8304152110 0x7f830406fbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51180/0 (socket says 172.21.15.78:51180) 2023-10-14T11:59:06.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.441+0000 7f8302ffd700 1 -- 172.21.15.78:0/1919083806 learned_addr learned my addr 172.21.15.78:0/1919083806 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:59:06.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.442+0000 7f83027fc700 1 --2- 172.21.15.78:0/1919083806 >> v2:172.21.15.195:3300/0 conn(0x7f830406d6b0 0x7f83040c6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:06.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.442+0000 7f83037fe700 1 -- 172.21.15.78:0/1919083806 >> v2:172.21.15.195:3300/0 conn(0x7f830406d6b0 msgr2=0x7f83040c6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:06.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.442+0000 7f83037fe700 1 --2- 172.21.15.78:0/1919083806 >> v2:172.21.15.195:3300/0 conn(0x7f830406d6b0 0x7f83040c6270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:06.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.442+0000 7f83037fe700 1 -- 172.21.15.78:0/1919083806 >> v1:172.21.15.78:6789/0 conn(0x7f8304152110 legacy=0x7f830406fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:06.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.442+0000 7f8301ffb700 1 -- 172.21.15.78:0/1919083806 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3983139919 0 0) 0x7f8304158450 con 0x7f8304152110 2023-10-14T11:59:06.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.442+0000 7f83037fe700 1 -- 172.21.15.78:0/1919083806 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8304158a10 con 0x7f83040c7340 2023-10-14T11:59:06.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.442+0000 7f83037fe700 1 --2- 172.21.15.78:0/1919083806 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83040c7340 0x7f83040c67b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f830406db40 tx=0x7f82fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf7889e9876d6d3c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:06.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.442+0000 7f8301ffb700 1 -- 172.21.15.78:0/1919083806 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82fc00e070 con 0x7f83040c7340 2023-10-14T11:59:06.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.442+0000 7f8301ffb700 1 -- 172.21.15.78:0/1919083806 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82fc00bbd0 con 0x7f83040c7340 2023-10-14T11:59:06.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.442+0000 7f8301ffb700 1 -- 172.21.15.78:0/1919083806 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82fc012430 con 0x7f83040c7340 2023-10-14T11:59:06.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.443+0000 7f83092b8700 1 -- 172.21.15.78:0/1919083806 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83040c7340 msgr2=0x7f83040c67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:06.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.443+0000 7f83092b8700 1 --2- 172.21.15.78:0/1919083806 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83040c7340 0x7f83040c67b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f830406db40 tx=0x7f82fc00b330 comp rx=0 tx=0).stop 2023-10-14T11:59:06.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.443+0000 7f83092b8700 1 -- 172.21.15.78:0/1919083806 shutdown_connections 2023-10-14T11:59:06.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.443+0000 7f83092b8700 1 --2- 172.21.15.78:0/1919083806 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83040c7340 0x7f83040c67b0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:06.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.443+0000 7f83092b8700 1 --2- 172.21.15.78:0/1919083806 >> v2:172.21.15.195:3300/0 conn(0x7f830406d6b0 0x7f83040c6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:06.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.443+0000 7f83092b8700 1 -- 172.21.15.78:0/1919083806 >> 172.21.15.78:0/1919083806 conn(0x7f83040c80d0 msgr2=0x7f83040c84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:06.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.443+0000 7f83092b8700 1 -- 172.21.15.78:0/1919083806 shutdown_connections 2023-10-14T11:59:06.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.443+0000 7f83092b8700 1 -- 172.21.15.78:0/1919083806 wait complete. 2023-10-14T11:59:06.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.444+0000 7f83092b8700 1 Processor -- start 2023-10-14T11:59:06.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.444+0000 7f83092b8700 1 -- start start 2023-10-14T11:59:06.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.445+0000 7f83092b8700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f830406d6b0 0x7f830406c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:06.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.445+0000 7f83092b8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83040c7340 0x7f830414e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:06.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.445+0000 7f83092b8700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8304152110 0x7f83041539d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:06.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.445+0000 7f83092b8700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83041ed6e0 con 0x7f83040c7340 2023-10-14T11:59:06.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.445+0000 7f83092b8700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83040c1f50 con 0x7f8304152110 2023-10-14T11:59:06.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.445+0000 7f83092b8700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83041ecdd0 con 0x7f830406d6b0 2023-10-14T11:59:06.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.445+0000 7f83027fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83040c7340 0x7f830414e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:06.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.445+0000 7f83027fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83040c7340 0x7f830414e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46312/0 (socket says 172.21.15.78:46312) 2023-10-14T11:59:06.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.445+0000 7f83027fc700 1 -- 172.21.15.78:0/2138566051 learned_addr learned my addr 172.21.15.78:0/2138566051 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:06.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.445+0000 7f8302ffd700 1 --2- 172.21.15.78:0/2138566051 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f830406d6b0 0x7f830406c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:06.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.445+0000 7f83037fe700 1 --2- 172.21.15.78:0/2138566051 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8304152110 0x7f83041539d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:06.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.445+0000 7f83027fc700 1 -- 172.21.15.78:0/2138566051 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f830406d6b0 msgr2=0x7f830406c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:06.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.445+0000 7f83027fc700 1 --2- 172.21.15.78:0/2138566051 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f830406d6b0 0x7f830406c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:06.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.445+0000 7f83027fc700 1 -- 172.21.15.78:0/2138566051 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8304152110 msgr2=0x7f83041539d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:06.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.445+0000 7f83027fc700 1 --2- 172.21.15.78:0/2138566051 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8304152110 0x7f83041539d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:06.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.445+0000 7f83027fc700 1 -- 172.21.15.78:0/2138566051 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82fc009a70 con 0x7f83040c7340 2023-10-14T11:59:06.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.445+0000 7f83037fe700 1 --2- 172.21.15.78:0/2138566051 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8304152110 0x7f83041539d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:59:06.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.445+0000 7f83027fc700 1 --2- 172.21.15.78:0/2138566051 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83040c7340 0x7f830414e660 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f82f400c7f0 tx=0x7f82f400cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:06.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.445+0000 7f82eb7fe700 1 -- 172.21.15.78:0/2138566051 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f4011800 con 0x7f83040c7340 2023-10-14T11:59:06.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.445+0000 7f82eb7fe700 1 -- 172.21.15.78:0/2138566051 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82f4011960 con 0x7f83040c7340 2023-10-14T11:59:06.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.445+0000 7f83092b8700 1 -- 172.21.15.78:0/2138566051 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83040bf4d0 con 0x7f83040c7340 2023-10-14T11:59:06.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.445+0000 7f82eb7fe700 1 -- 172.21.15.78:0/2138566051 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f4011cd0 con 0x7f83040c7340 2023-10-14T11:59:06.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.445+0000 7f83092b8700 1 -- 172.21.15.78:0/2138566051 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83041ed300 con 0x7f83040c7340 2023-10-14T11:59:06.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.446+0000 7f83092b8700 1 -- 172.21.15.78:0/2138566051 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83041ed820 con 0x7f83040c7340 2023-10-14T11:59:06.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.447+0000 7f82eb7fe700 1 -- 172.21.15.78:0/2138566051 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f82f4010350 con 0x7f83040c7340 2023-10-14T11:59:06.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.447+0000 7f82eb7fe700 1 --2- 172.21.15.78:0/2138566051 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f82f006af00 0x7f82f006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:06.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.447+0000 7f82eb7fe700 1 -- 172.21.15.78:0/2138566051 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f82f4089090 con 0x7f83040c7340 2023-10-14T11:59:06.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.447+0000 7f8302ffd700 1 --2- 172.21.15.78:0/2138566051 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f82f006af00 0x7f82f006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:06.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.447+0000 7f8302ffd700 1 --2- 172.21.15.78:0/2138566051 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f82f006af00 0x7f82f006d3c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f82f8007fd0 tx=0x7f82f8007de0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:06.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.451+0000 7f82eb7fe700 1 -- 172.21.15.78:0/2138566051 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f83041ed820 con 0x7f83040c7340 2023-10-14T11:59:06.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.637+0000 7f83092b8700 1 -- 172.21.15.78:0/2138566051 --> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8304158cf0 con 0x7f82f006af00 2023-10-14T11:59:06.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.644+0000 7f82eb7fe700 1 -- 172.21.15.78:0/2138566051 <== mgr.4104 v2:172.21.15.78:6832/3171890074 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+263555 (secure 0 0 0) 0x7f82f8001370 con 0x7f82f006af00 2023-10-14T11:59:06.650 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:59:06.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.647+0000 7f83092b8700 1 -- 172.21.15.78:0/2138566051 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f82f006af00 msgr2=0x7f82f006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:06.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.647+0000 7f83092b8700 1 --2- 172.21.15.78:0/2138566051 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f82f006af00 0x7f82f006d3c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f82f8007fd0 tx=0x7f82f8007de0 comp rx=0 tx=0).stop 2023-10-14T11:59:06.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.647+0000 7f83092b8700 1 -- 172.21.15.78:0/2138566051 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83040c7340 msgr2=0x7f830414e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:06.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.647+0000 7f83092b8700 1 --2- 172.21.15.78:0/2138566051 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83040c7340 0x7f830414e660 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f82f400c7f0 tx=0x7f82f400cb00 comp rx=0 tx=0).stop 2023-10-14T11:59:06.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.648+0000 7f83092b8700 1 -- 172.21.15.78:0/2138566051 shutdown_connections 2023-10-14T11:59:06.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.648+0000 7f83092b8700 1 --2- 172.21.15.78:0/2138566051 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f82f006af00 0x7f82f006d3c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:06.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.648+0000 7f83092b8700 1 --2- 172.21.15.78:0/2138566051 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8304152110 0x7f83041539d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:06.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.648+0000 7f83092b8700 1 --2- 172.21.15.78:0/2138566051 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f830406d6b0 0x7f830406c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:06.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.648+0000 7f83092b8700 1 --2- 172.21.15.78:0/2138566051 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83040c7340 0x7f830414e660 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:06.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.648+0000 7f83092b8700 1 -- 172.21.15.78:0/2138566051 >> 172.21.15.78:0/2138566051 conn(0x7f83040c80d0 msgr2=0x7f83040c3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:06.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.648+0000 7f83092b8700 1 -- 172.21.15.78:0/2138566051 shutdown_connections 2023-10-14T11:59:06.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.648+0000 7f83092b8700 1 -- 172.21.15.78:0/2138566051 wait complete. 2023-10-14T11:59:06.655 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2023-10-14T11:59:06.661 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":54,"stamp":"2023-10-14T11:59:04.975403+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":73,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218932,"kb_used_data":2764,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755788,"statfs":{"total":773094113280,"available":772869926912,"internally_reserved":0,"allocated":2830336,"data_stored":2193229,"data_compressed":65244,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.817370"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.143338+0000","last_change":"2023-10-14T11:59:02.143338+0000","last_active":"2023-10-14T11:59:02.143338+0000","last_peered":"2023-10-14T11:59:02.143338+0000","last_clean":"2023-10-14T11:59:02.143338+0000","last_became_active":"2023-10-14T11:59:02.143049+0000","last_became_peered":"2023-10-14T11:59:02.143049+0000","last_unstale":"2023-10-14T11:59:02.143338+0000","last_undegraded":"2023-10-14T11:59:02.143338+0000","last_fullsized":"2023-10-14T11:59:02.143338+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:43:53.205090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":4,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.139597+0000","last_change":"2023-10-14T11:59:02.138579+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.139597+0000","last_undegraded":"2023-10-14T11:59:02.139597+0000","last_fullsized":"2023-10-14T11:59:02.139597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:35:59.785182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.150827+0000","last_change":"2023-10-14T11:59:02.144594+0000","last_active":"2023-10-14T11:59:02.150827+0000","last_peered":"2023-10-14T11:59:02.150827+0000","last_clean":"2023-10-14T11:59:02.150827+0000","last_became_active":"2023-10-14T11:59:02.144322+0000","last_became_peered":"2023-10-14T11:59:02.144322+0000","last_unstale":"2023-10-14T11:59:02.150827+0000","last_undegraded":"2023-10-14T11:59:02.150827+0000","last_fullsized":"2023-10-14T11:59:02.150827+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:08:25.866669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.146281+0000","last_change":"2023-10-14T11:59:02.146281+0000","last_active":"2023-10-14T11:59:02.146281+0000","last_peered":"2023-10-14T11:59:02.146281+0000","last_clean":"2023-10-14T11:59:02.146281+0000","last_became_active":"2023-10-14T11:59:02.146015+0000","last_became_peered":"2023-10-14T11:59:02.146015+0000","last_unstale":"2023-10-14T11:59:02.146281+0000","last_undegraded":"2023-10-14T11:59:02.146281+0000","last_fullsized":"2023-10-14T11:59:02.146281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:54:28.849250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.140518+0000","last_change":"2023-10-14T11:59:02.140518+0000","last_active":"2023-10-14T11:59:02.140518+0000","last_peered":"2023-10-14T11:59:02.140518+0000","last_clean":"2023-10-14T11:59:02.140518+0000","last_became_active":"2023-10-14T11:59:02.140238+0000","last_became_peered":"2023-10-14T11:59:02.140238+0000","last_unstale":"2023-10-14T11:59:02.140518+0000","last_undegraded":"2023-10-14T11:59:02.140518+0000","last_fullsized":"2023-10-14T11:59:02.140518+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:43:35.020618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.136431+0000","last_change":"2023-10-14T11:59:02.136431+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.136431+0000","last_undegraded":"2023-10-14T11:59:02.136431+0000","last_fullsized":"2023-10-14T11:59:02.136431+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:08:25.866669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.135963+0000","last_change":"2023-10-14T11:59:02.135963+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.135963+0000","last_undegraded":"2023-10-14T11:59:02.135963+0000","last_fullsized":"2023-10-14T11:59:02.135963+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:05:38.808781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.135986+0000","last_change":"2023-10-14T11:59:02.135986+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.135986+0000","last_undegraded":"2023-10-14T11:59:02.135986+0000","last_fullsized":"2023-10-14T11:59:02.135986+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:57:18.507681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.132676+0000","last_change":"2023-10-14T11:59:02.132676+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.132676+0000","last_undegraded":"2023-10-14T11:59:02.132676+0000","last_fullsized":"2023-10-14T11:59:02.132676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:20:02.525878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.133724+0000","last_change":"2023-10-14T11:59:02.133724+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.133724+0000","last_undegraded":"2023-10-14T11:59:02.133724+0000","last_fullsized":"2023-10-14T11:59:02.133724+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:15:00.593091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.143474+0000","last_change":"2023-10-14T11:59:02.143474+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.143474+0000","last_undegraded":"2023-10-14T11:59:02.143474+0000","last_fullsized":"2023-10-14T11:59:02.143474+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:07:15.007617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.143255+0000","last_change":"2023-10-14T11:59:02.139674+0000","last_active":"2023-10-14T11:59:02.143255+0000","last_peered":"2023-10-14T11:59:02.143255+0000","last_clean":"2023-10-14T11:59:02.143255+0000","last_became_active":"2023-10-14T11:59:02.139404+0000","last_became_peered":"2023-10-14T11:59:02.139404+0000","last_unstale":"2023-10-14T11:59:02.143255+0000","last_undegraded":"2023-10-14T11:59:02.143255+0000","last_fullsized":"2023-10-14T11:59:02.143255+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:30:41.965448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.145057+0000","last_change":"2023-10-14T11:59:02.145057+0000","last_active":"2023-10-14T11:59:02.145057+0000","last_peered":"2023-10-14T11:59:02.145057+0000","last_clean":"2023-10-14T11:59:02.145057+0000","last_became_active":"2023-10-14T11:59:02.144776+0000","last_became_peered":"2023-10-14T11:59:02.144776+0000","last_unstale":"2023-10-14T11:59:02.145057+0000","last_undegraded":"2023-10-14T11:59:02.145057+0000","last_fullsized":"2023-10-14T11:59:02.145057+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:03:39.032281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.137158+0000","last_change":"2023-10-14T11:59:02.137158+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.137158+0000","last_undegraded":"2023-10-14T11:59:02.137158+0000","last_fullsized":"2023-10-14T11:59:02.137158+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:12:09.205507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.140788+0000","last_change":"2023-10-14T11:59:02.134707+0000","last_active":"2023-10-14T11:59:02.140788+0000","last_peered":"2023-10-14T11:59:02.140788+0000","last_clean":"2023-10-14T11:59:02.140788+0000","last_became_active":"2023-10-14T11:59:02.134424+0000","last_became_peered":"2023-10-14T11:59:02.134424+0000","last_unstale":"2023-10-14T11:59:02.140788+0000","last_undegraded":"2023-10-14T11:59:02.140788+0000","last_fullsized":"2023-10-14T11:59:02.140788+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:14.681110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.138148+0000","last_change":"2023-10-14T11:59:02.138148+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.138148+0000","last_undegraded":"2023-10-14T11:59:02.138148+0000","last_fullsized":"2023-10-14T11:59:02.138148+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:27:07.788319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.136715+0000","last_change":"2023-10-14T11:59:02.136715+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.136715+0000","last_undegraded":"2023-10-14T11:59:02.136715+0000","last_fullsized":"2023-10-14T11:59:02.136715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:30:41.965448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.135337+0000","last_change":"2023-10-14T11:59:02.135337+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.135337+0000","last_undegraded":"2023-10-14T11:59:02.135337+0000","last_fullsized":"2023-10-14T11:59:02.135337+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:05:38.808781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.141545+0000","last_change":"2023-10-14T11:59:02.141545+0000","last_active":"2023-10-14T11:59:02.141545+0000","last_peered":"2023-10-14T11:59:02.141545+0000","last_clean":"2023-10-14T11:59:02.141545+0000","last_became_active":"2023-10-14T11:59:02.141281+0000","last_became_peered":"2023-10-14T11:59:02.141281+0000","last_unstale":"2023-10-14T11:59:02.141545+0000","last_undegraded":"2023-10-14T11:59:02.141545+0000","last_fullsized":"2023-10-14T11:59:02.141545+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:26.625395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.145736+0000","last_change":"2023-10-14T11:59:02.145736+0000","last_active":"2023-10-14T11:59:02.145736+0000","last_peered":"2023-10-14T11:59:02.145736+0000","last_clean":"2023-10-14T11:59:02.145736+0000","last_became_active":"2023-10-14T11:59:02.145465+0000","last_became_peered":"2023-10-14T11:59:02.145465+0000","last_unstale":"2023-10-14T11:59:02.145736+0000","last_undegraded":"2023-10-14T11:59:02.145736+0000","last_fullsized":"2023-10-14T11:59:02.145736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:08:45.915417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.132115+0000","last_change":"2023-10-14T11:59:02.132115+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.132115+0000","last_undegraded":"2023-10-14T11:59:02.132115+0000","last_fullsized":"2023-10-14T11:59:02.132115+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:53:44.574661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.132815+0000","last_change":"2023-10-14T11:59:02.132815+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.132815+0000","last_undegraded":"2023-10-14T11:59:02.132815+0000","last_fullsized":"2023-10-14T11:59:02.132815+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:39:49.443148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.137732+0000","last_change":"2023-10-14T11:59:02.137732+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.137732+0000","last_undegraded":"2023-10-14T11:59:02.137732+0000","last_fullsized":"2023-10-14T11:59:02.137732+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:54:28.849250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.150417+0000","last_change":"2023-10-14T11:59:02.149221+0000","last_active":"2023-10-14T11:59:02.150417+0000","last_peered":"2023-10-14T11:59:02.150417+0000","last_clean":"2023-10-14T11:59:02.150417+0000","last_became_active":"2023-10-14T11:59:02.148952+0000","last_became_peered":"2023-10-14T11:59:02.148952+0000","last_unstale":"2023-10-14T11:59:02.150417+0000","last_undegraded":"2023-10-14T11:59:02.150417+0000","last_fullsized":"2023-10-14T11:59:02.150417+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:58:07.847931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.146060+0000","last_change":"2023-10-14T11:59:02.146060+0000","last_active":"2023-10-14T11:59:02.146060+0000","last_peered":"2023-10-14T11:59:02.146060+0000","last_clean":"2023-10-14T11:59:02.146060+0000","last_became_active":"2023-10-14T11:59:02.145805+0000","last_became_peered":"2023-10-14T11:59:02.145805+0000","last_unstale":"2023-10-14T11:59:02.146060+0000","last_undegraded":"2023-10-14T11:59:02.146060+0000","last_fullsized":"2023-10-14T11:59:02.146060+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:39:49.443148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.132003+0000","last_change":"2023-10-14T11:59:02.132003+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.132003+0000","last_undegraded":"2023-10-14T11:59:02.132003+0000","last_fullsized":"2023-10-14T11:59:02.132003+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:54:28.849250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.146857+0000","last_change":"2023-10-14T11:59:02.146857+0000","last_active":"2023-10-14T11:59:02.146857+0000","last_peered":"2023-10-14T11:59:02.146857+0000","last_clean":"2023-10-14T11:59:02.146857+0000","last_became_active":"2023-10-14T11:59:02.146627+0000","last_became_peered":"2023-10-14T11:59:02.146627+0000","last_unstale":"2023-10-14T11:59:02.146857+0000","last_undegraded":"2023-10-14T11:59:02.146857+0000","last_fullsized":"2023-10-14T11:59:02.146857+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:30:41.965448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.149774+0000","last_change":"2023-10-14T11:59:02.143747+0000","last_active":"2023-10-14T11:59:02.149774+0000","last_peered":"2023-10-14T11:59:02.149774+0000","last_clean":"2023-10-14T11:59:02.149774+0000","last_became_active":"2023-10-14T11:59:02.143421+0000","last_became_peered":"2023-10-14T11:59:02.143421+0000","last_unstale":"2023-10-14T11:59:02.149774+0000","last_undegraded":"2023-10-14T11:59:02.149774+0000","last_fullsized":"2023-10-14T11:59:02.149774+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:27:07.788319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.139034+0000","last_change":"2023-10-14T11:59:02.139034+0000","last_active":"2023-10-14T11:59:02.139034+0000","last_peered":"2023-10-14T11:59:02.139034+0000","last_clean":"2023-10-14T11:59:02.139034+0000","last_became_active":"2023-10-14T11:59:02.138730+0000","last_became_peered":"2023-10-14T11:59:02.138730+0000","last_unstale":"2023-10-14T11:59:02.139034+0000","last_undegraded":"2023-10-14T11:59:02.139034+0000","last_fullsized":"2023-10-14T11:59:02.139034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:48:37.789205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.136790+0000","last_change":"2023-10-14T11:59:02.136790+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.136790+0000","last_undegraded":"2023-10-14T11:59:02.136790+0000","last_fullsized":"2023-10-14T11:59:02.136790+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:58:07.847931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.131455+0000","last_change":"2023-10-14T11:59:02.131455+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.131455+0000","last_undegraded":"2023-10-14T11:59:02.131455+0000","last_fullsized":"2023-10-14T11:59:02.131455+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:05:58.896548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.138555+0000","last_change":"2023-10-14T11:59:02.138555+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.138555+0000","last_undegraded":"2023-10-14T11:59:02.138555+0000","last_fullsized":"2023-10-14T11:59:02.138555+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:08:25.866669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.136873+0000","last_change":"2023-10-14T11:59:02.136873+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.136873+0000","last_undegraded":"2023-10-14T11:59:02.136873+0000","last_fullsized":"2023-10-14T11:59:02.136873+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:20:02.525878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.133827+0000","last_change":"2023-10-14T11:59:01.170225+0000","last_active":"2023-10-14T11:59:02.133827+0000","last_peered":"2023-10-14T11:59:02.133827+0000","last_clean":"2023-10-14T11:59:02.133827+0000","last_became_active":"2023-10-14T11:59:01.169927+0000","last_became_peered":"2023-10-14T11:59:01.169927+0000","last_unstale":"2023-10-14T11:59:02.133827+0000","last_undegraded":"2023-10-14T11:59:02.133827+0000","last_fullsized":"2023-10-14T11:59:02.133827+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:25.623744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.139277+0000","last_change":"2023-10-14T11:59:02.139277+0000","last_active":"2023-10-14T11:59:02.139277+0000","last_peered":"2023-10-14T11:59:02.139277+0000","last_clean":"2023-10-14T11:59:02.139277+0000","last_became_active":"2023-10-14T11:59:02.138985+0000","last_became_peered":"2023-10-14T11:59:02.138985+0000","last_unstale":"2023-10-14T11:59:02.139277+0000","last_undegraded":"2023-10-14T11:59:02.139277+0000","last_fullsized":"2023-10-14T11:59:02.139277+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:15:00.593091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.137138+0000","last_change":"2023-10-14T11:59:01.178406+0000","last_active":"2023-10-14T11:59:02.137138+0000","last_peered":"2023-10-14T11:59:02.137138+0000","last_clean":"2023-10-14T11:59:02.137138+0000","last_became_active":"2023-10-14T11:59:01.178136+0000","last_became_peered":"2023-10-14T11:59:01.178136+0000","last_unstale":"2023-10-14T11:59:02.137138+0000","last_undegraded":"2023-10-14T11:59:02.137138+0000","last_fullsized":"2023-10-14T11:59:02.137138+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:43:52.203439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.139738+0000","last_change":"2023-10-14T11:59:02.139738+0000","last_active":"2023-10-14T11:59:02.139738+0000","last_peered":"2023-10-14T11:59:02.139738+0000","last_clean":"2023-10-14T11:59:02.139738+0000","last_became_active":"2023-10-14T11:59:02.139431+0000","last_became_peered":"2023-10-14T11:59:02.139431+0000","last_unstale":"2023-10-14T11:59:02.139738+0000","last_undegraded":"2023-10-14T11:59:02.139738+0000","last_fullsized":"2023-10-14T11:59:02.139738+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:41:47.404184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.133347+0000","last_change":"2023-10-14T11:59:01.179865+0000","last_active":"2023-10-14T11:59:02.133347+0000","last_peered":"2023-10-14T11:59:02.133347+0000","last_clean":"2023-10-14T11:59:02.133347+0000","last_became_active":"2023-10-14T11:59:01.178404+0000","last_became_peered":"2023-10-14T11:59:01.178404+0000","last_unstale":"2023-10-14T11:59:02.133347+0000","last_undegraded":"2023-10-14T11:59:02.133347+0000","last_fullsized":"2023-10-14T11:59:02.133347+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:30:40.963797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.146195+0000","last_change":"2023-10-14T11:59:02.146195+0000","last_active":"2023-10-14T11:59:02.146195+0000","last_peered":"2023-10-14T11:59:02.146195+0000","last_clean":"2023-10-14T11:59:02.146195+0000","last_became_active":"2023-10-14T11:59:02.144184+0000","last_became_peered":"2023-10-14T11:59:02.144184+0000","last_unstale":"2023-10-14T11:59:02.146195+0000","last_undegraded":"2023-10-14T11:59:02.146195+0000","last_fullsized":"2023-10-14T11:59:02.146195+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:43:53.205090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.135778+0000","last_change":"2023-10-14T11:59:01.185486+0000","last_active":"2023-10-14T11:59:02.135778+0000","last_peered":"2023-10-14T11:59:02.135778+0000","last_clean":"2023-10-14T11:59:02.135778+0000","last_became_active":"2023-10-14T11:59:01.185200+0000","last_became_peered":"2023-10-14T11:59:01.185200+0000","last_unstale":"2023-10-14T11:59:02.135778+0000","last_undegraded":"2023-10-14T11:59:02.135778+0000","last_fullsized":"2023-10-14T11:59:02.135778+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:48:36.787554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.139052+0000","last_change":"2023-10-14T11:59:02.139052+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.139052+0000","last_undegraded":"2023-10-14T11:59:02.139052+0000","last_fullsized":"2023-10-14T11:59:02.139052+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:36:52.548523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.132966+0000","last_change":"2023-10-14T11:59:01.178774+0000","last_active":"2023-10-14T11:59:02.132966+0000","last_peered":"2023-10-14T11:59:02.132966+0000","last_clean":"2023-10-14T11:59:02.132966+0000","last_became_active":"2023-10-14T11:59:01.178378+0000","last_became_peered":"2023-10-14T11:59:01.178378+0000","last_unstale":"2023-10-14T11:59:02.132966+0000","last_undegraded":"2023-10-14T11:59:02.132966+0000","last_fullsized":"2023-10-14T11:59:02.132966+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:25.623744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.149841+0000","last_change":"2023-10-14T11:59:02.143614+0000","last_active":"2023-10-14T11:59:02.149841+0000","last_peered":"2023-10-14T11:59:02.149841+0000","last_clean":"2023-10-14T11:59:02.149841+0000","last_became_active":"2023-10-14T11:59:02.143332+0000","last_became_peered":"2023-10-14T11:59:02.143332+0000","last_unstale":"2023-10-14T11:59:02.149841+0000","last_undegraded":"2023-10-14T11:59:02.149841+0000","last_fullsized":"2023-10-14T11:59:02.149841+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:12:09.205507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.133277+0000","last_change":"2023-10-14T11:59:01.178778+0000","last_active":"2023-10-14T11:59:02.133277+0000","last_peered":"2023-10-14T11:59:02.133277+0000","last_clean":"2023-10-14T11:59:02.133277+0000","last_became_active":"2023-10-14T11:59:01.178440+0000","last_became_peered":"2023-10-14T11:59:01.178440+0000","last_unstale":"2023-10-14T11:59:02.133277+0000","last_undegraded":"2023-10-14T11:59:02.133277+0000","last_fullsized":"2023-10-14T11:59:02.133277+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:18:16.224262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.139185+0000","last_change":"2023-10-14T11:59:02.139185+0000","last_active":"2023-10-14T11:59:02.139185+0000","last_peered":"2023-10-14T11:59:02.139185+0000","last_clean":"2023-10-14T11:59:02.139185+0000","last_became_active":"2023-10-14T11:59:02.138862+0000","last_became_peered":"2023-10-14T11:59:02.138862+0000","last_unstale":"2023-10-14T11:59:02.139185+0000","last_undegraded":"2023-10-14T11:59:02.139185+0000","last_fullsized":"2023-10-14T11:59:02.139185+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:48:45.563805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.132488+0000","last_change":"2023-10-14T11:59:01.175662+0000","last_active":"2023-10-14T11:59:02.132488+0000","last_peered":"2023-10-14T11:59:02.132488+0000","last_clean":"2023-10-14T11:59:02.132488+0000","last_became_active":"2023-10-14T11:59:01.175405+0000","last_became_peered":"2023-10-14T11:59:01.175405+0000","last_unstale":"2023-10-14T11:59:02.132488+0000","last_undegraded":"2023-10-14T11:59:02.132488+0000","last_fullsized":"2023-10-14T11:59:02.132488+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:57:17.506030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.143559+0000","last_change":"2023-10-14T11:59:02.143559+0000","last_active":"2023-10-14T11:59:02.143559+0000","last_peered":"2023-10-14T11:59:02.143559+0000","last_clean":"2023-10-14T11:59:02.143559+0000","last_became_active":"2023-10-14T11:59:02.143303+0000","last_became_peered":"2023-10-14T11:59:02.143303+0000","last_unstale":"2023-10-14T11:59:02.143559+0000","last_undegraded":"2023-10-14T11:59:02.143559+0000","last_fullsized":"2023-10-14T11:59:02.143559+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:20:02.525878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.132854+0000","last_change":"2023-10-14T11:59:01.167233+0000","last_active":"2023-10-14T11:59:02.132854+0000","last_peered":"2023-10-14T11:59:02.132854+0000","last_clean":"2023-10-14T11:59:02.132854+0000","last_became_active":"2023-10-14T11:59:01.166905+0000","last_became_peered":"2023-10-14T11:59:01.166905+0000","last_unstale":"2023-10-14T11:59:02.132854+0000","last_undegraded":"2023-10-14T11:59:02.132854+0000","last_fullsized":"2023-10-14T11:59:02.132854+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:43:52.203439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.152247+0000","last_change":"2023-10-14T11:59:02.150665+0000","last_active":"2023-10-14T11:59:02.152247+0000","last_peered":"2023-10-14T11:59:02.152247+0000","last_clean":"2023-10-14T11:59:02.152247+0000","last_became_active":"2023-10-14T11:59:02.150384+0000","last_became_peered":"2023-10-14T11:59:02.150384+0000","last_unstale":"2023-10-14T11:59:02.152247+0000","last_undegraded":"2023-10-14T11:59:02.152247+0000","last_fullsized":"2023-10-14T11:59:02.152247+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:07:15.007617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.133851+0000","last_change":"2023-10-14T11:59:01.179314+0000","last_active":"2023-10-14T11:59:02.133851+0000","last_peered":"2023-10-14T11:59:02.133851+0000","last_clean":"2023-10-14T11:59:02.133851+0000","last_became_active":"2023-10-14T11:59:01.179019+0000","last_became_peered":"2023-10-14T11:59:01.179019+0000","last_unstale":"2023-10-14T11:59:02.133851+0000","last_undegraded":"2023-10-14T11:59:02.133851+0000","last_fullsized":"2023-10-14T11:59:02.133851+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:27:06.786668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.137973+0000","last_change":"2023-10-14T11:59:02.137973+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.137973+0000","last_undegraded":"2023-10-14T11:59:02.137973+0000","last_fullsized":"2023-10-14T11:59:02.137973+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:36:52.548523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.137264+0000","last_change":"2023-10-14T11:59:01.177873+0000","last_active":"2023-10-14T11:59:02.137264+0000","last_peered":"2023-10-14T11:59:02.137264+0000","last_clean":"2023-10-14T11:59:02.137264+0000","last_became_active":"2023-10-14T11:59:01.177594+0000","last_became_peered":"2023-10-14T11:59:01.177594+0000","last_unstale":"2023-10-14T11:59:02.137264+0000","last_undegraded":"2023-10-14T11:59:02.137264+0000","last_fullsized":"2023-10-14T11:59:02.137264+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:25.623744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.132187+0000","last_change":"2023-10-14T11:59:02.132187+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.132187+0000","last_undegraded":"2023-10-14T11:59:02.132187+0000","last_fullsized":"2023-10-14T11:59:02.132187+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:47:13.361893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.135909+0000","last_change":"2023-10-14T11:59:01.178143+0000","last_active":"2023-10-14T11:59:02.135909+0000","last_peered":"2023-10-14T11:59:02.135909+0000","last_clean":"2023-10-14T11:59:02.135909+0000","last_became_active":"2023-10-14T11:59:01.177736+0000","last_became_peered":"2023-10-14T11:59:01.177736+0000","last_unstale":"2023-10-14T11:59:02.135909+0000","last_undegraded":"2023-10-14T11:59:02.135909+0000","last_fullsized":"2023-10-14T11:59:02.135909+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:35:58.783531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.150847+0000","last_change":"2023-10-14T11:59:02.145816+0000","last_active":"2023-10-14T11:59:02.150847+0000","last_peered":"2023-10-14T11:59:02.150847+0000","last_clean":"2023-10-14T11:59:02.150847+0000","last_became_active":"2023-10-14T11:59:02.145548+0000","last_became_peered":"2023-10-14T11:59:02.145548+0000","last_unstale":"2023-10-14T11:59:02.150847+0000","last_undegraded":"2023-10-14T11:59:02.150847+0000","last_fullsized":"2023-10-14T11:59:02.150847+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:13:03.743718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.133733+0000","last_change":"2023-10-14T11:59:01.180187+0000","last_active":"2023-10-14T11:59:02.133733+0000","last_peered":"2023-10-14T11:59:02.133733+0000","last_clean":"2023-10-14T11:59:02.133733+0000","last_became_active":"2023-10-14T11:59:01.179909+0000","last_became_peered":"2023-10-14T11:59:01.179909+0000","last_unstale":"2023-10-14T11:59:02.133733+0000","last_undegraded":"2023-10-14T11:59:02.133733+0000","last_fullsized":"2023-10-14T11:59:02.133733+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:58:06.846280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.143696+0000","last_change":"2023-10-14T11:59:02.143696+0000","last_active":"2023-10-14T11:59:02.143696+0000","last_peered":"2023-10-14T11:59:02.143696+0000","last_clean":"2023-10-14T11:59:02.143696+0000","last_became_active":"2023-10-14T11:59:02.143443+0000","last_became_peered":"2023-10-14T11:59:02.143443+0000","last_unstale":"2023-10-14T11:59:02.143696+0000","last_undegraded":"2023-10-14T11:59:02.143696+0000","last_fullsized":"2023-10-14T11:59:02.143696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:47:13.361893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.134332+0000","last_change":"2023-10-14T11:59:01.165901+0000","last_active":"2023-10-14T11:59:02.134332+0000","last_peered":"2023-10-14T11:59:02.134332+0000","last_clean":"2023-10-14T11:59:02.134332+0000","last_became_active":"2023-10-14T11:59:01.165597+0000","last_became_peered":"2023-10-14T11:59:01.165597+0000","last_unstale":"2023-10-14T11:59:02.134332+0000","last_undegraded":"2023-10-14T11:59:02.134332+0000","last_fullsized":"2023-10-14T11:59:02.134332+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:05:37.807130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.141427+0000","last_change":"2023-10-14T11:59:02.141427+0000","last_active":"2023-10-14T11:59:02.141427+0000","last_peered":"2023-10-14T11:59:02.141427+0000","last_clean":"2023-10-14T11:59:02.141427+0000","last_became_active":"2023-10-14T11:59:02.141166+0000","last_became_peered":"2023-10-14T11:59:02.141166+0000","last_unstale":"2023-10-14T11:59:02.141427+0000","last_undegraded":"2023-10-14T11:59:02.141427+0000","last_fullsized":"2023-10-14T11:59:02.141427+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:18:17.225913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.136387+0000","last_change":"2023-10-14T11:59:01.172037+0000","last_active":"2023-10-14T11:59:02.136387+0000","last_peered":"2023-10-14T11:59:02.136387+0000","last_clean":"2023-10-14T11:59:02.136387+0000","last_became_active":"2023-10-14T11:59:01.170760+0000","last_became_peered":"2023-10-14T11:59:01.170760+0000","last_unstale":"2023-10-14T11:59:02.136387+0000","last_undegraded":"2023-10-14T11:59:02.136387+0000","last_fullsized":"2023-10-14T11:59:02.136387+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:57:17.506030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.138925+0000","last_change":"2023-10-14T11:59:02.138925+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.138925+0000","last_undegraded":"2023-10-14T11:59:02.138925+0000","last_fullsized":"2023-10-14T11:59:02.138925+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:12:09.205507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.132642+0000","last_change":"2023-10-14T11:59:01.178960+0000","last_active":"2023-10-14T11:59:02.132642+0000","last_peered":"2023-10-14T11:59:02.132642+0000","last_clean":"2023-10-14T11:59:02.132642+0000","last_became_active":"2023-10-14T11:59:01.178691+0000","last_became_peered":"2023-10-14T11:59:01.178691+0000","last_unstale":"2023-10-14T11:59:02.132642+0000","last_undegraded":"2023-10-14T11:59:02.132642+0000","last_fullsized":"2023-10-14T11:59:02.132642+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:13.679459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.140181+0000","last_change":"2023-10-14T11:59:02.140181+0000","last_active":"2023-10-14T11:59:02.140181+0000","last_peered":"2023-10-14T11:59:02.140181+0000","last_clean":"2023-10-14T11:59:02.140181+0000","last_became_active":"2023-10-14T11:59:02.139382+0000","last_became_peered":"2023-10-14T11:59:02.139382+0000","last_unstale":"2023-10-14T11:59:02.140181+0000","last_undegraded":"2023-10-14T11:59:02.140181+0000","last_fullsized":"2023-10-14T11:59:02.140181+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:20:02.525878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.138336+0000","last_change":"2023-10-14T11:59:01.178769+0000","last_active":"2023-10-14T11:59:02.138336+0000","last_peered":"2023-10-14T11:59:02.138336+0000","last_clean":"2023-10-14T11:59:02.138336+0000","last_became_active":"2023-10-14T11:59:01.178288+0000","last_became_peered":"2023-10-14T11:59:01.178288+0000","last_unstale":"2023-10-14T11:59:02.138336+0000","last_undegraded":"2023-10-14T11:59:02.138336+0000","last_fullsized":"2023-10-14T11:59:02.138336+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:07:14.005966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.132368+0000","last_change":"2023-10-14T11:59:02.132368+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.132368+0000","last_undegraded":"2023-10-14T11:59:02.132368+0000","last_fullsized":"2023-10-14T11:59:02.132368+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:05:58.896548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.134614+0000","last_change":"2023-10-14T11:59:01.183630+0000","last_active":"2023-10-14T11:59:02.134614+0000","last_peered":"2023-10-14T11:59:02.134614+0000","last_clean":"2023-10-14T11:59:02.134614+0000","last_became_active":"2023-10-14T11:59:01.183408+0000","last_became_peered":"2023-10-14T11:59:01.183408+0000","last_unstale":"2023-10-14T11:59:02.134614+0000","last_undegraded":"2023-10-14T11:59:02.134614+0000","last_fullsized":"2023-10-14T11:59:02.134614+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:25.623744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.145545+0000","last_change":"2023-10-14T11:59:02.141878+0000","last_active":"2023-10-14T11:59:02.145545+0000","last_peered":"2023-10-14T11:59:02.145545+0000","last_clean":"2023-10-14T11:59:02.145545+0000","last_became_active":"2023-10-14T11:59:02.141229+0000","last_became_peered":"2023-10-14T11:59:02.141229+0000","last_unstale":"2023-10-14T11:59:02.145545+0000","last_undegraded":"2023-10-14T11:59:02.145545+0000","last_fullsized":"2023-10-14T11:59:02.145545+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:11:18.803472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.132715+0000","last_change":"2023-10-14T11:59:01.165942+0000","last_active":"2023-10-14T11:59:02.132715+0000","last_peered":"2023-10-14T11:59:02.132715+0000","last_clean":"2023-10-14T11:59:02.132715+0000","last_became_active":"2023-10-14T11:59:01.165646+0000","last_became_peered":"2023-10-14T11:59:01.165646+0000","last_unstale":"2023-10-14T11:59:02.132715+0000","last_undegraded":"2023-10-14T11:59:02.132715+0000","last_fullsized":"2023-10-14T11:59:02.132715+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:14:59.591440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.147269+0000","last_change":"2023-10-14T11:59:02.147269+0000","last_active":"2023-10-14T11:59:02.147269+0000","last_peered":"2023-10-14T11:59:02.147269+0000","last_clean":"2023-10-14T11:59:02.147269+0000","last_became_active":"2023-10-14T11:59:02.147005+0000","last_became_peered":"2023-10-14T11:59:02.147005+0000","last_unstale":"2023-10-14T11:59:02.147269+0000","last_undegraded":"2023-10-14T11:59:02.147269+0000","last_fullsized":"2023-10-14T11:59:02.147269+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:18:17.225913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.131810+0000","last_change":"2023-10-14T11:59:01.183080+0000","last_active":"2023-10-14T11:59:02.131810+0000","last_peered":"2023-10-14T11:59:02.131810+0000","last_clean":"2023-10-14T11:59:02.131810+0000","last_became_active":"2023-10-14T11:59:01.182786+0000","last_became_peered":"2023-10-14T11:59:01.182786+0000","last_unstale":"2023-10-14T11:59:02.131810+0000","last_undegraded":"2023-10-14T11:59:02.131810+0000","last_fullsized":"2023-10-14T11:59:02.131810+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:05:27.508197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.146618+0000","last_change":"2023-10-14T11:59:02.146618+0000","last_active":"2023-10-14T11:59:02.146618+0000","last_peered":"2023-10-14T11:59:02.146618+0000","last_clean":"2023-10-14T11:59:02.146618+0000","last_became_active":"2023-10-14T11:59:02.146352+0000","last_became_peered":"2023-10-14T11:59:02.146352+0000","last_unstale":"2023-10-14T11:59:02.146618+0000","last_undegraded":"2023-10-14T11:59:02.146618+0000","last_fullsized":"2023-10-14T11:59:02.146618+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:48:45.563805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.132063+0000","last_change":"2023-10-14T11:59:01.178771+0000","last_active":"2023-10-14T11:59:02.132063+0000","last_peered":"2023-10-14T11:59:02.132063+0000","last_clean":"2023-10-14T11:59:02.132063+0000","last_became_active":"2023-10-14T11:59:01.178416+0000","last_became_peered":"2023-10-14T11:59:01.178416+0000","last_unstale":"2023-10-14T11:59:02.132063+0000","last_undegraded":"2023-10-14T11:59:02.132063+0000","last_fullsized":"2023-10-14T11:59:02.132063+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:11:46.659890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.145624+0000","last_change":"2023-10-14T11:59:02.145624+0000","last_active":"2023-10-14T11:59:02.145624+0000","last_peered":"2023-10-14T11:59:02.145624+0000","last_clean":"2023-10-14T11:59:02.145624+0000","last_became_active":"2023-10-14T11:59:02.145381+0000","last_became_peered":"2023-10-14T11:59:02.145381+0000","last_unstale":"2023-10-14T11:59:02.145624+0000","last_undegraded":"2023-10-14T11:59:02.145624+0000","last_fullsized":"2023-10-14T11:59:02.145624+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:21:57.196717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.137214+0000","last_change":"2023-10-14T11:59:01.177556+0000","last_active":"2023-10-14T11:59:02.137214+0000","last_peered":"2023-10-14T11:59:02.137214+0000","last_clean":"2023-10-14T11:59:02.137214+0000","last_became_active":"2023-10-14T11:59:01.177313+0000","last_became_peered":"2023-10-14T11:59:01.177313+0000","last_unstale":"2023-10-14T11:59:02.137214+0000","last_undegraded":"2023-10-14T11:59:02.137214+0000","last_fullsized":"2023-10-14T11:59:02.137214+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:27:02.969220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.141409+0000","last_change":"2023-10-14T11:59:02.141409+0000","last_active":"2023-10-14T11:59:02.141409+0000","last_peered":"2023-10-14T11:59:02.141409+0000","last_clean":"2023-10-14T11:59:02.141409+0000","last_became_active":"2023-10-14T11:59:02.141163+0000","last_became_peered":"2023-10-14T11:59:02.141163+0000","last_unstale":"2023-10-14T11:59:02.141409+0000","last_undegraded":"2023-10-14T11:59:02.141409+0000","last_fullsized":"2023-10-14T11:59:02.141409+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:54:06.957363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.133688+0000","last_change":"2023-10-14T11:59:01.168240+0000","last_active":"2023-10-14T11:59:02.133688+0000","last_peered":"2023-10-14T11:59:02.133688+0000","last_clean":"2023-10-14T11:59:02.133688+0000","last_became_active":"2023-10-14T11:59:01.167944+0000","last_became_peered":"2023-10-14T11:59:01.167944+0000","last_unstale":"2023-10-14T11:59:02.133688+0000","last_undegraded":"2023-10-14T11:59:02.133688+0000","last_fullsized":"2023-10-14T11:59:02.133688+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:27:02.969220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":4,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.140710+0000","last_change":"2023-10-14T11:59:02.133062+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.140710+0000","last_undegraded":"2023-10-14T11:59:02.140710+0000","last_fullsized":"2023-10-14T11:59:02.140710+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:40:14.734059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.133269+0000","last_change":"2023-10-14T11:59:01.176530+0000","last_active":"2023-10-14T11:59:02.133269+0000","last_peered":"2023-10-14T11:59:02.133269+0000","last_clean":"2023-10-14T11:59:02.133269+0000","last_became_active":"2023-10-14T11:59:01.176193+0000","last_became_peered":"2023-10-14T11:59:01.176193+0000","last_unstale":"2023-10-14T11:59:02.133269+0000","last_undegraded":"2023-10-14T11:59:02.133269+0000","last_fullsized":"2023-10-14T11:59:02.133269+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:54:27.847599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.136372+0000","last_change":"2023-10-14T11:59:02.136372+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.136372+0000","last_undegraded":"2023-10-14T11:59:02.136372+0000","last_fullsized":"2023-10-14T11:59:02.136372+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:14.681110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.130535+0000","last_change":"2023-10-14T11:59:01.178157+0000","last_active":"2023-10-14T11:59:02.130535+0000","last_peered":"2023-10-14T11:59:02.130535+0000","last_clean":"2023-10-14T11:59:02.130535+0000","last_became_active":"2023-10-14T11:59:01.177863+0000","last_became_peered":"2023-10-14T11:59:01.177863+0000","last_unstale":"2023-10-14T11:59:02.130535+0000","last_undegraded":"2023-10-14T11:59:02.130535+0000","last_fullsized":"2023-10-14T11:59:02.130535+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:57:17.506030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.143662+0000","last_change":"2023-10-14T11:59:02.143662+0000","last_active":"2023-10-14T11:59:02.143662+0000","last_peered":"2023-10-14T11:59:02.143662+0000","last_clean":"2023-10-14T11:59:02.143662+0000","last_became_active":"2023-10-14T11:59:02.143456+0000","last_became_peered":"2023-10-14T11:59:02.143456+0000","last_unstale":"2023-10-14T11:59:02.143662+0000","last_undegraded":"2023-10-14T11:59:02.143662+0000","last_fullsized":"2023-10-14T11:59:02.143662+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:33:27.410202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.134465+0000","last_change":"2023-10-14T11:59:01.181546+0000","last_active":"2023-10-14T11:59:02.134465+0000","last_peered":"2023-10-14T11:59:02.134465+0000","last_clean":"2023-10-14T11:59:02.134465+0000","last_became_active":"2023-10-14T11:59:01.181255+0000","last_became_peered":"2023-10-14T11:59:01.181255+0000","last_unstale":"2023-10-14T11:59:02.134465+0000","last_undegraded":"2023-10-14T11:59:02.134465+0000","last_fullsized":"2023-10-14T11:59:02.134465+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:27:02.969220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.145565+0000","last_change":"2023-10-14T11:59:02.142119+0000","last_active":"2023-10-14T11:59:02.145565+0000","last_peered":"2023-10-14T11:59:02.145565+0000","last_clean":"2023-10-14T11:59:02.145565+0000","last_became_active":"2023-10-14T11:59:02.141586+0000","last_became_peered":"2023-10-14T11:59:02.141586+0000","last_unstale":"2023-10-14T11:59:02.145565+0000","last_undegraded":"2023-10-14T11:59:02.145565+0000","last_fullsized":"2023-10-14T11:59:02.145565+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:36:52.548523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.137387+0000","last_change":"2023-10-14T11:59:01.180897+0000","last_active":"2023-10-14T11:59:02.137387+0000","last_peered":"2023-10-14T11:59:02.137387+0000","last_clean":"2023-10-14T11:59:02.137387+0000","last_became_active":"2023-10-14T11:59:01.180524+0000","last_became_peered":"2023-10-14T11:59:01.180524+0000","last_unstale":"2023-10-14T11:59:02.137387+0000","last_undegraded":"2023-10-14T11:59:02.137387+0000","last_fullsized":"2023-10-14T11:59:02.137387+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:18:16.224262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.136894+0000","last_change":"2023-10-14T11:59:02.136894+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.136894+0000","last_undegraded":"2023-10-14T11:59:02.136894+0000","last_fullsized":"2023-10-14T11:59:02.136894+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:15:00.593091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.136362+0000","last_change":"2023-10-14T11:59:01.183591+0000","last_active":"2023-10-14T11:59:02.136362+0000","last_peered":"2023-10-14T11:59:02.136362+0000","last_clean":"2023-10-14T11:59:02.136362+0000","last_became_active":"2023-10-14T11:59:01.183019+0000","last_became_peered":"2023-10-14T11:59:01.183019+0000","last_unstale":"2023-10-14T11:59:02.136362+0000","last_undegraded":"2023-10-14T11:59:02.136362+0000","last_fullsized":"2023-10-14T11:59:02.136362+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:05:37.807130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.133356+0000","last_change":"2023-10-14T11:59:02.133356+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.133356+0000","last_undegraded":"2023-10-14T11:59:02.133356+0000","last_fullsized":"2023-10-14T11:59:02.133356+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:39:49.443148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.137080+0000","last_change":"2023-10-14T11:59:01.177539+0000","last_active":"2023-10-14T11:59:02.137080+0000","last_peered":"2023-10-14T11:59:02.137080+0000","last_clean":"2023-10-14T11:59:02.137080+0000","last_became_active":"2023-10-14T11:59:01.177255+0000","last_became_peered":"2023-10-14T11:59:01.177255+0000","last_unstale":"2023-10-14T11:59:02.137080+0000","last_undegraded":"2023-10-14T11:59:02.137080+0000","last_fullsized":"2023-10-14T11:59:02.137080+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:39:48.441497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.137750+0000","last_change":"2023-10-14T11:59:02.137750+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.137750+0000","last_undegraded":"2023-10-14T11:59:02.137750+0000","last_fullsized":"2023-10-14T11:59:02.137750+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:47:13.361893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.136562+0000","last_change":"2023-10-14T11:59:01.182837+0000","last_active":"2023-10-14T11:59:02.136562+0000","last_peered":"2023-10-14T11:59:02.136562+0000","last_clean":"2023-10-14T11:59:02.136562+0000","last_became_active":"2023-10-14T11:59:01.182236+0000","last_became_peered":"2023-10-14T11:59:01.182236+0000","last_unstale":"2023-10-14T11:59:02.136562+0000","last_undegraded":"2023-10-14T11:59:02.136562+0000","last_fullsized":"2023-10-14T11:59:02.136562+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:13.679459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":4,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.138516+0000","last_change":"2023-10-14T11:59:02.138103+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.138516+0000","last_undegraded":"2023-10-14T11:59:02.138516+0000","last_fullsized":"2023-10-14T11:59:02.138516+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:13:03.743718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.133395+0000","last_change":"2023-10-14T11:59:01.168077+0000","last_active":"2023-10-14T11:59:02.133395+0000","last_peered":"2023-10-14T11:59:02.133395+0000","last_clean":"2023-10-14T11:59:02.133395+0000","last_became_active":"2023-10-14T11:59:01.167803+0000","last_became_peered":"2023-10-14T11:59:01.167803+0000","last_unstale":"2023-10-14T11:59:02.133395+0000","last_undegraded":"2023-10-14T11:59:02.133395+0000","last_fullsized":"2023-10-14T11:59:02.133395+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:47:12.360242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.139387+0000","last_change":"2023-10-14T11:59:02.139387+0000","last_active":"2023-10-14T11:59:02.139387+0000","last_peered":"2023-10-14T11:59:02.139387+0000","last_clean":"2023-10-14T11:59:02.139387+0000","last_became_active":"2023-10-14T11:59:02.139117+0000","last_became_peered":"2023-10-14T11:59:02.139117+0000","last_unstale":"2023-10-14T11:59:02.139387+0000","last_undegraded":"2023-10-14T11:59:02.139387+0000","last_fullsized":"2023-10-14T11:59:02.139387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:27:03.970871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.138469+0000","last_change":"2023-10-14T11:59:01.179436+0000","last_active":"2023-10-14T11:59:02.138469+0000","last_peered":"2023-10-14T11:59:02.138469+0000","last_clean":"2023-10-14T11:59:02.138469+0000","last_became_active":"2023-10-14T11:59:01.179109+0000","last_became_peered":"2023-10-14T11:59:01.179109+0000","last_unstale":"2023-10-14T11:59:02.138469+0000","last_undegraded":"2023-10-14T11:59:02.138469+0000","last_fullsized":"2023-10-14T11:59:02.138469+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:11:17.801821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.143298+0000","last_change":"2023-10-14T11:59:02.143298+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.143298+0000","last_undegraded":"2023-10-14T11:59:02.143298+0000","last_fullsized":"2023-10-14T11:59:02.143298+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:13:03.743718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.131329+0000","last_change":"2023-10-14T11:59:01.169175+0000","last_active":"2023-10-14T11:59:02.131329+0000","last_peered":"2023-10-14T11:59:02.131329+0000","last_clean":"2023-10-14T11:59:02.131329+0000","last_became_active":"2023-10-14T11:59:01.168873+0000","last_became_peered":"2023-10-14T11:59:01.168873+0000","last_unstale":"2023-10-14T11:59:02.131329+0000","last_undegraded":"2023-10-14T11:59:02.131329+0000","last_fullsized":"2023-10-14T11:59:02.131329+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:53:43.573010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":172,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-14T11:59:01.174640+0000","last_change":"2023-10-14T11:58:10.616726+0000","last_active":"2023-10-14T11:59:01.174640+0000","last_peered":"2023-10-14T11:59:01.174640+0000","last_clean":"2023-10-14T11:59:01.174640+0000","last_became_active":"2023-10-14T11:58:10.615628+0000","last_became_peered":"2023-10-14T11:58:10.615628+0000","last_unstale":"2023-10-14T11:59:01.174640+0000","last_undegraded":"2023-10-14T11:59:01.174640+0000","last_fullsized":"2023-10-14T11:59:01.174640+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:09.001265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:09.001265+0000","last_clean_scrub_stamp":"2023-10-14T11:58:09.001265+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:22:50.496050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65244,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574862,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":716079,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574862,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":125711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84399999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574862,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":132516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]}]},{"osd":0,"up_from":13,"seq":55834574861,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":401408,"data_stored":716079,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574861,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":125711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.254}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574862,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":125711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574862,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":125711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574862,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":125711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.311}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T11:59:06.668 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T11:59:06.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.820+0000 7fa92e959700 1 Processor -- start 2023-10-14T11:59:06.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.820+0000 7fa92e959700 1 -- start start 2023-10-14T11:59:06.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.821+0000 7fa92e959700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa92806ab40 0x7fa92806f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:06.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.821+0000 7fa92e959700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa9281554e0 0x7fa92806af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:06.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.821+0000 7fa92e959700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9280c49f0 con 0x7fa9280c0350 2023-10-14T11:59:06.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.821+0000 7fa92e959700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9280726a0 con 0x7fa9281554e0 2023-10-14T11:59:06.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.821+0000 7fa92e959700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa928072980 con 0x7fa92806ab40 2023-10-14T11:59:06.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.821+0000 7fa9277fe700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa9281554e0 0x7fa92806af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:06.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.821+0000 7fa9277fe700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa9281554e0 0x7fa92806af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46318/0 (socket says 172.21.15.78:46318) 2023-10-14T11:59:06.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.821+0000 7fa9277fe700 1 -- 172.21.15.78:0/2612765816 learned_addr learned my addr 172.21.15.78:0/2612765816 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:06.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.821+0000 7fa927fff700 1 --2- 172.21.15.78:0/2612765816 >> v2:172.21.15.195:3300/0 conn(0x7fa92806ab40 0x7fa92806f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:06.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.821+0000 7fa9277fe700 1 -- 172.21.15.78:0/2612765816 >> v2:172.21.15.195:3300/0 conn(0x7fa92806ab40 msgr2=0x7fa92806f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:06.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.821+0000 7fa9277fe700 1 --2- 172.21.15.78:0/2612765816 >> v2:172.21.15.195:3300/0 conn(0x7fa92806ab40 0x7fa92806f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:06.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.821+0000 7fa9277fe700 1 -- 172.21.15.78:0/2612765816 >> v1:172.21.15.195:6789/0 conn(0x7fa9280c0350 legacy=0x7fa92813d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:06.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.821+0000 7fa9277fe700 1 -- 172.21.15.78:0/2612765816 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa928154a00 con 0x7fa9281554e0 2023-10-14T11:59:06.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.821+0000 7fa9277fe700 1 --2- 172.21.15.78:0/2612765816 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9281554e0 0x7fa92806af20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa928072bc0 tx=0x7fa918009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da28c571ecd65c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:06.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.821+0000 7fa926ffd700 1 -- 172.21.15.78:0/2612765816 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa918011830 con 0x7fa9281554e0 2023-10-14T11:59:06.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.822+0000 7fa926ffd700 1 -- 172.21.15.78:0/2612765816 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa918011990 con 0x7fa9281554e0 2023-10-14T11:59:06.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.822+0000 7fa926ffd700 1 -- 172.21.15.78:0/2612765816 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa918011d20 con 0x7fa9281554e0 2023-10-14T11:59:06.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.822+0000 7fa92e959700 1 -- 172.21.15.78:0/2612765816 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9281554e0 msgr2=0x7fa92806af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:06.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.822+0000 7fa92e959700 1 --2- 172.21.15.78:0/2612765816 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9281554e0 0x7fa92806af20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa928072bc0 tx=0x7fa918009bd0 comp rx=0 tx=0).stop 2023-10-14T11:59:06.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.822+0000 7fa92e959700 1 -- 172.21.15.78:0/2612765816 shutdown_connections 2023-10-14T11:59:06.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.822+0000 7fa92e959700 1 --2- 172.21.15.78:0/2612765816 >> v2:172.21.15.195:3300/0 conn(0x7fa92806ab40 0x7fa92806f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:06.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.822+0000 7fa92e959700 1 --2- 172.21.15.78:0/2612765816 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9281554e0 0x7fa92806af20 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:06.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.822+0000 7fa92e959700 1 -- 172.21.15.78:0/2612765816 >> 172.21.15.78:0/2612765816 conn(0x7fa9280c3ac0 msgr2=0x7fa9280c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:06.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.822+0000 7fa92e959700 1 -- 172.21.15.78:0/2612765816 shutdown_connections 2023-10-14T11:59:06.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.822+0000 7fa92e959700 1 -- 172.21.15.78:0/2612765816 wait complete. 2023-10-14T11:59:06.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.823+0000 7fa92e959700 1 Processor -- start 2023-10-14T11:59:06.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.823+0000 7fa92e959700 1 -- start start 2023-10-14T11:59:06.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.824+0000 7fa92e959700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa92806ab40 0x7fa928151990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:06.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.824+0000 7fa92e959700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9280c0350 0x7fa9281ecde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:06.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.824+0000 7fa92e959700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa9281554e0 0x7fa9281ed320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:06.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.824+0000 7fa92e959700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9281e9870 con 0x7fa9280c0350 2023-10-14T11:59:06.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.824+0000 7fa92e959700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9281ea820 con 0x7fa92806ab40 2023-10-14T11:59:06.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.824+0000 7fa92e959700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9281e9a60 con 0x7fa9281554e0 2023-10-14T11:59:06.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.824+0000 7fa9277fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9280c0350 0x7fa9281ecde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:06.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.824+0000 7fa9277fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9280c0350 0x7fa9281ecde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46328/0 (socket says 172.21.15.78:46328) 2023-10-14T11:59:06.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.824+0000 7fa9277fe700 1 -- 172.21.15.78:0/2870177867 learned_addr learned my addr 172.21.15.78:0/2870177867 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:06.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.824+0000 7fa927fff700 1 --2- 172.21.15.78:0/2870177867 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa92806ab40 0x7fa928151990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:06.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.824+0000 7fa92cef6700 1 --2- 172.21.15.78:0/2870177867 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa9281554e0 0x7fa9281ed320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:06.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.824+0000 7fa9277fe700 1 -- 172.21.15.78:0/2870177867 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa9281554e0 msgr2=0x7fa9281ed320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:06.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.824+0000 7fa9277fe700 1 --2- 172.21.15.78:0/2870177867 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa9281554e0 0x7fa9281ed320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:06.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.824+0000 7fa9277fe700 1 -- 172.21.15.78:0/2870177867 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa92806ab40 msgr2=0x7fa928151990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:06.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.824+0000 7fa9277fe700 1 --2- 172.21.15.78:0/2870177867 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa92806ab40 0x7fa928151990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:06.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.824+0000 7fa9277fe700 1 -- 172.21.15.78:0/2870177867 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa918009550 con 0x7fa9280c0350 2023-10-14T11:59:06.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.825+0000 7fa9277fe700 1 --2- 172.21.15.78:0/2870177867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9280c0350 0x7fa9281ecde0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa918014040 tx=0x7fa918004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:06.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.825+0000 7fa924ff9700 1 -- 172.21.15.78:0/2870177867 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9180036a0 con 0x7fa9280c0350 2023-10-14T11:59:06.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.825+0000 7fa924ff9700 1 -- 172.21.15.78:0/2870177867 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa91801e070 con 0x7fa9280c0350 2023-10-14T11:59:06.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.825+0000 7fa92e959700 1 -- 172.21.15.78:0/2870177867 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9281e9d40 con 0x7fa9280c0350 2023-10-14T11:59:06.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.825+0000 7fa924ff9700 1 -- 172.21.15.78:0/2870177867 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa918003a60 con 0x7fa9280c0350 2023-10-14T11:59:06.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.825+0000 7fa92e959700 1 -- 172.21.15.78:0/2870177867 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9281ea300 con 0x7fa9280c0350 2023-10-14T11:59:06.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.826+0000 7fa92e959700 1 -- 172.21.15.78:0/2870177867 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa92806b440 con 0x7fa9280c0350 2023-10-14T11:59:06.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.826+0000 7fa924ff9700 1 -- 172.21.15.78:0/2870177867 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa918003bc0 con 0x7fa9280c0350 2023-10-14T11:59:06.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.826+0000 7fa924ff9700 1 --2- 172.21.15.78:0/2870177867 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa91406af00 0x7fa91406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:06.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.827+0000 7fa924ff9700 1 -- 172.21.15.78:0/2870177867 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7fa91808a4c0 con 0x7fa9280c0350 2023-10-14T11:59:06.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.827+0000 7fa927fff700 1 --2- 172.21.15.78:0/2870177867 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa91406af00 0x7fa91406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:06.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.827+0000 7fa927fff700 1 --2- 172.21.15.78:0/2870177867 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa91406af00 0x7fa91406d3c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa9281e76b0 tx=0x7fa91c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:06.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:06.831+0000 7fa924ff9700 1 -- 172.21.15.78:0/2870177867 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa92806b440 con 0x7fa9280c0350 2023-10-14T11:59:07.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.016+0000 7fa92e959700 1 -- 172.21.15.78:0/2870177867 --> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa928154fc0 con 0x7fa91406af00 2023-10-14T11:59:07.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.023+0000 7fa924ff9700 1 -- 172.21.15.78:0/2870177867 <== mgr.4104 v2:172.21.15.78:6832/3171890074 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+263555 (secure 0 0 0) 0x7fa91c0117b0 con 0x7fa91406af00 2023-10-14T11:59:07.029 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:59:07.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.026+0000 7fa92e959700 1 -- 172.21.15.78:0/2870177867 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa91406af00 msgr2=0x7fa91406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:07.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.026+0000 7fa92e959700 1 --2- 172.21.15.78:0/2870177867 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa91406af00 0x7fa91406d3c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa9281e76b0 tx=0x7fa91c008040 comp rx=0 tx=0).stop 2023-10-14T11:59:07.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.026+0000 7fa92e959700 1 -- 172.21.15.78:0/2870177867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9280c0350 msgr2=0x7fa9281ecde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:07.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.026+0000 7fa92e959700 1 --2- 172.21.15.78:0/2870177867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9280c0350 0x7fa9281ecde0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa918014040 tx=0x7fa918004750 comp rx=0 tx=0).stop 2023-10-14T11:59:07.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.026+0000 7fa92e959700 1 -- 172.21.15.78:0/2870177867 shutdown_connections 2023-10-14T11:59:07.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.026+0000 7fa92e959700 1 --2- 172.21.15.78:0/2870177867 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa9281554e0 0x7fa9281ed320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:07.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.026+0000 7fa92e959700 1 --2- 172.21.15.78:0/2870177867 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa91406af00 0x7fa91406d3c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:07.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.026+0000 7fa92e959700 1 --2- 172.21.15.78:0/2870177867 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa92806ab40 0x7fa928151990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:07.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.026+0000 7fa92e959700 1 --2- 172.21.15.78:0/2870177867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9280c0350 0x7fa9281ecde0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:07.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.026+0000 7fa92e959700 1 -- 172.21.15.78:0/2870177867 >> 172.21.15.78:0/2870177867 conn(0x7fa9280c3ac0 msgr2=0x7fa92806ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:07.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.027+0000 7fa92e959700 1 -- 172.21.15.78:0/2870177867 shutdown_connections 2023-10-14T11:59:07.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.027+0000 7fa92e959700 1 -- 172.21.15.78:0/2870177867 wait complete. 2023-10-14T11:59:07.033 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2023-10-14T11:59:07.041 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":55,"stamp":"2023-10-14T11:59:06.975942+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":73,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218932,"kb_used_data":2764,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755788,"statfs":{"total":773094113280,"available":772869926912,"internally_reserved":0,"allocated":2830336,"data_stored":2193229,"data_compressed":65244,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001480"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.143338+0000","last_change":"2023-10-14T11:59:02.143338+0000","last_active":"2023-10-14T11:59:02.143338+0000","last_peered":"2023-10-14T11:59:02.143338+0000","last_clean":"2023-10-14T11:59:02.143338+0000","last_became_active":"2023-10-14T11:59:02.143049+0000","last_became_peered":"2023-10-14T11:59:02.143049+0000","last_unstale":"2023-10-14T11:59:02.143338+0000","last_undegraded":"2023-10-14T11:59:02.143338+0000","last_fullsized":"2023-10-14T11:59:02.143338+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:43:53.205090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":4,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.139597+0000","last_change":"2023-10-14T11:59:02.138579+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.139597+0000","last_undegraded":"2023-10-14T11:59:02.139597+0000","last_fullsized":"2023-10-14T11:59:02.139597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:35:59.785182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.150827+0000","last_change":"2023-10-14T11:59:02.144594+0000","last_active":"2023-10-14T11:59:02.150827+0000","last_peered":"2023-10-14T11:59:02.150827+0000","last_clean":"2023-10-14T11:59:02.150827+0000","last_became_active":"2023-10-14T11:59:02.144322+0000","last_became_peered":"2023-10-14T11:59:02.144322+0000","last_unstale":"2023-10-14T11:59:02.150827+0000","last_undegraded":"2023-10-14T11:59:02.150827+0000","last_fullsized":"2023-10-14T11:59:02.150827+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:08:25.866669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.146281+0000","last_change":"2023-10-14T11:59:02.146281+0000","last_active":"2023-10-14T11:59:02.146281+0000","last_peered":"2023-10-14T11:59:02.146281+0000","last_clean":"2023-10-14T11:59:02.146281+0000","last_became_active":"2023-10-14T11:59:02.146015+0000","last_became_peered":"2023-10-14T11:59:02.146015+0000","last_unstale":"2023-10-14T11:59:02.146281+0000","last_undegraded":"2023-10-14T11:59:02.146281+0000","last_fullsized":"2023-10-14T11:59:02.146281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:54:28.849250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.140518+0000","last_change":"2023-10-14T11:59:02.140518+0000","last_active":"2023-10-14T11:59:02.140518+0000","last_peered":"2023-10-14T11:59:02.140518+0000","last_clean":"2023-10-14T11:59:02.140518+0000","last_became_active":"2023-10-14T11:59:02.140238+0000","last_became_peered":"2023-10-14T11:59:02.140238+0000","last_unstale":"2023-10-14T11:59:02.140518+0000","last_undegraded":"2023-10-14T11:59:02.140518+0000","last_fullsized":"2023-10-14T11:59:02.140518+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:43:35.020618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.136431+0000","last_change":"2023-10-14T11:59:02.136431+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.136431+0000","last_undegraded":"2023-10-14T11:59:02.136431+0000","last_fullsized":"2023-10-14T11:59:02.136431+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:08:25.866669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.135963+0000","last_change":"2023-10-14T11:59:02.135963+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.135963+0000","last_undegraded":"2023-10-14T11:59:02.135963+0000","last_fullsized":"2023-10-14T11:59:02.135963+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:05:38.808781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.135986+0000","last_change":"2023-10-14T11:59:02.135986+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.135986+0000","last_undegraded":"2023-10-14T11:59:02.135986+0000","last_fullsized":"2023-10-14T11:59:02.135986+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:57:18.507681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.132676+0000","last_change":"2023-10-14T11:59:02.132676+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.132676+0000","last_undegraded":"2023-10-14T11:59:02.132676+0000","last_fullsized":"2023-10-14T11:59:02.132676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:20:02.525878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.133724+0000","last_change":"2023-10-14T11:59:02.133724+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.133724+0000","last_undegraded":"2023-10-14T11:59:02.133724+0000","last_fullsized":"2023-10-14T11:59:02.133724+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:15:00.593091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.143474+0000","last_change":"2023-10-14T11:59:02.143474+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.143474+0000","last_undegraded":"2023-10-14T11:59:02.143474+0000","last_fullsized":"2023-10-14T11:59:02.143474+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:07:15.007617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.143255+0000","last_change":"2023-10-14T11:59:02.139674+0000","last_active":"2023-10-14T11:59:02.143255+0000","last_peered":"2023-10-14T11:59:02.143255+0000","last_clean":"2023-10-14T11:59:02.143255+0000","last_became_active":"2023-10-14T11:59:02.139404+0000","last_became_peered":"2023-10-14T11:59:02.139404+0000","last_unstale":"2023-10-14T11:59:02.143255+0000","last_undegraded":"2023-10-14T11:59:02.143255+0000","last_fullsized":"2023-10-14T11:59:02.143255+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:30:41.965448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.145057+0000","last_change":"2023-10-14T11:59:02.145057+0000","last_active":"2023-10-14T11:59:02.145057+0000","last_peered":"2023-10-14T11:59:02.145057+0000","last_clean":"2023-10-14T11:59:02.145057+0000","last_became_active":"2023-10-14T11:59:02.144776+0000","last_became_peered":"2023-10-14T11:59:02.144776+0000","last_unstale":"2023-10-14T11:59:02.145057+0000","last_undegraded":"2023-10-14T11:59:02.145057+0000","last_fullsized":"2023-10-14T11:59:02.145057+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:03:39.032281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.137158+0000","last_change":"2023-10-14T11:59:02.137158+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.137158+0000","last_undegraded":"2023-10-14T11:59:02.137158+0000","last_fullsized":"2023-10-14T11:59:02.137158+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:12:09.205507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.140788+0000","last_change":"2023-10-14T11:59:02.134707+0000","last_active":"2023-10-14T11:59:02.140788+0000","last_peered":"2023-10-14T11:59:02.140788+0000","last_clean":"2023-10-14T11:59:02.140788+0000","last_became_active":"2023-10-14T11:59:02.134424+0000","last_became_peered":"2023-10-14T11:59:02.134424+0000","last_unstale":"2023-10-14T11:59:02.140788+0000","last_undegraded":"2023-10-14T11:59:02.140788+0000","last_fullsized":"2023-10-14T11:59:02.140788+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:14.681110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.138148+0000","last_change":"2023-10-14T11:59:02.138148+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.138148+0000","last_undegraded":"2023-10-14T11:59:02.138148+0000","last_fullsized":"2023-10-14T11:59:02.138148+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:27:07.788319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.136715+0000","last_change":"2023-10-14T11:59:02.136715+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.136715+0000","last_undegraded":"2023-10-14T11:59:02.136715+0000","last_fullsized":"2023-10-14T11:59:02.136715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:30:41.965448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.135337+0000","last_change":"2023-10-14T11:59:02.135337+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.135337+0000","last_undegraded":"2023-10-14T11:59:02.135337+0000","last_fullsized":"2023-10-14T11:59:02.135337+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:05:38.808781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.141545+0000","last_change":"2023-10-14T11:59:02.141545+0000","last_active":"2023-10-14T11:59:02.141545+0000","last_peered":"2023-10-14T11:59:02.141545+0000","last_clean":"2023-10-14T11:59:02.141545+0000","last_became_active":"2023-10-14T11:59:02.141281+0000","last_became_peered":"2023-10-14T11:59:02.141281+0000","last_unstale":"2023-10-14T11:59:02.141545+0000","last_undegraded":"2023-10-14T11:59:02.141545+0000","last_fullsized":"2023-10-14T11:59:02.141545+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:26.625395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.145736+0000","last_change":"2023-10-14T11:59:02.145736+0000","last_active":"2023-10-14T11:59:02.145736+0000","last_peered":"2023-10-14T11:59:02.145736+0000","last_clean":"2023-10-14T11:59:02.145736+0000","last_became_active":"2023-10-14T11:59:02.145465+0000","last_became_peered":"2023-10-14T11:59:02.145465+0000","last_unstale":"2023-10-14T11:59:02.145736+0000","last_undegraded":"2023-10-14T11:59:02.145736+0000","last_fullsized":"2023-10-14T11:59:02.145736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:08:45.915417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.132115+0000","last_change":"2023-10-14T11:59:02.132115+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.132115+0000","last_undegraded":"2023-10-14T11:59:02.132115+0000","last_fullsized":"2023-10-14T11:59:02.132115+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:53:44.574661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.132815+0000","last_change":"2023-10-14T11:59:02.132815+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.132815+0000","last_undegraded":"2023-10-14T11:59:02.132815+0000","last_fullsized":"2023-10-14T11:59:02.132815+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:39:49.443148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.137732+0000","last_change":"2023-10-14T11:59:02.137732+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.137732+0000","last_undegraded":"2023-10-14T11:59:02.137732+0000","last_fullsized":"2023-10-14T11:59:02.137732+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:54:28.849250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.150417+0000","last_change":"2023-10-14T11:59:02.149221+0000","last_active":"2023-10-14T11:59:02.150417+0000","last_peered":"2023-10-14T11:59:02.150417+0000","last_clean":"2023-10-14T11:59:02.150417+0000","last_became_active":"2023-10-14T11:59:02.148952+0000","last_became_peered":"2023-10-14T11:59:02.148952+0000","last_unstale":"2023-10-14T11:59:02.150417+0000","last_undegraded":"2023-10-14T11:59:02.150417+0000","last_fullsized":"2023-10-14T11:59:02.150417+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:58:07.847931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.146060+0000","last_change":"2023-10-14T11:59:02.146060+0000","last_active":"2023-10-14T11:59:02.146060+0000","last_peered":"2023-10-14T11:59:02.146060+0000","last_clean":"2023-10-14T11:59:02.146060+0000","last_became_active":"2023-10-14T11:59:02.145805+0000","last_became_peered":"2023-10-14T11:59:02.145805+0000","last_unstale":"2023-10-14T11:59:02.146060+0000","last_undegraded":"2023-10-14T11:59:02.146060+0000","last_fullsized":"2023-10-14T11:59:02.146060+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:39:49.443148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.132003+0000","last_change":"2023-10-14T11:59:02.132003+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.132003+0000","last_undegraded":"2023-10-14T11:59:02.132003+0000","last_fullsized":"2023-10-14T11:59:02.132003+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:54:28.849250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.146857+0000","last_change":"2023-10-14T11:59:02.146857+0000","last_active":"2023-10-14T11:59:02.146857+0000","last_peered":"2023-10-14T11:59:02.146857+0000","last_clean":"2023-10-14T11:59:02.146857+0000","last_became_active":"2023-10-14T11:59:02.146627+0000","last_became_peered":"2023-10-14T11:59:02.146627+0000","last_unstale":"2023-10-14T11:59:02.146857+0000","last_undegraded":"2023-10-14T11:59:02.146857+0000","last_fullsized":"2023-10-14T11:59:02.146857+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:30:41.965448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.149774+0000","last_change":"2023-10-14T11:59:02.143747+0000","last_active":"2023-10-14T11:59:02.149774+0000","last_peered":"2023-10-14T11:59:02.149774+0000","last_clean":"2023-10-14T11:59:02.149774+0000","last_became_active":"2023-10-14T11:59:02.143421+0000","last_became_peered":"2023-10-14T11:59:02.143421+0000","last_unstale":"2023-10-14T11:59:02.149774+0000","last_undegraded":"2023-10-14T11:59:02.149774+0000","last_fullsized":"2023-10-14T11:59:02.149774+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:27:07.788319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.139034+0000","last_change":"2023-10-14T11:59:02.139034+0000","last_active":"2023-10-14T11:59:02.139034+0000","last_peered":"2023-10-14T11:59:02.139034+0000","last_clean":"2023-10-14T11:59:02.139034+0000","last_became_active":"2023-10-14T11:59:02.138730+0000","last_became_peered":"2023-10-14T11:59:02.138730+0000","last_unstale":"2023-10-14T11:59:02.139034+0000","last_undegraded":"2023-10-14T11:59:02.139034+0000","last_fullsized":"2023-10-14T11:59:02.139034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:48:37.789205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.136790+0000","last_change":"2023-10-14T11:59:02.136790+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.136790+0000","last_undegraded":"2023-10-14T11:59:02.136790+0000","last_fullsized":"2023-10-14T11:59:02.136790+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:58:07.847931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.131455+0000","last_change":"2023-10-14T11:59:02.131455+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.131455+0000","last_undegraded":"2023-10-14T11:59:02.131455+0000","last_fullsized":"2023-10-14T11:59:02.131455+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:05:58.896548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.138555+0000","last_change":"2023-10-14T11:59:02.138555+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.138555+0000","last_undegraded":"2023-10-14T11:59:02.138555+0000","last_fullsized":"2023-10-14T11:59:02.138555+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:08:25.866669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.136873+0000","last_change":"2023-10-14T11:59:02.136873+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.136873+0000","last_undegraded":"2023-10-14T11:59:02.136873+0000","last_fullsized":"2023-10-14T11:59:02.136873+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:20:02.525878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.133827+0000","last_change":"2023-10-14T11:59:01.170225+0000","last_active":"2023-10-14T11:59:02.133827+0000","last_peered":"2023-10-14T11:59:02.133827+0000","last_clean":"2023-10-14T11:59:02.133827+0000","last_became_active":"2023-10-14T11:59:01.169927+0000","last_became_peered":"2023-10-14T11:59:01.169927+0000","last_unstale":"2023-10-14T11:59:02.133827+0000","last_undegraded":"2023-10-14T11:59:02.133827+0000","last_fullsized":"2023-10-14T11:59:02.133827+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:25.623744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.139277+0000","last_change":"2023-10-14T11:59:02.139277+0000","last_active":"2023-10-14T11:59:02.139277+0000","last_peered":"2023-10-14T11:59:02.139277+0000","last_clean":"2023-10-14T11:59:02.139277+0000","last_became_active":"2023-10-14T11:59:02.138985+0000","last_became_peered":"2023-10-14T11:59:02.138985+0000","last_unstale":"2023-10-14T11:59:02.139277+0000","last_undegraded":"2023-10-14T11:59:02.139277+0000","last_fullsized":"2023-10-14T11:59:02.139277+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:15:00.593091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.137138+0000","last_change":"2023-10-14T11:59:01.178406+0000","last_active":"2023-10-14T11:59:02.137138+0000","last_peered":"2023-10-14T11:59:02.137138+0000","last_clean":"2023-10-14T11:59:02.137138+0000","last_became_active":"2023-10-14T11:59:01.178136+0000","last_became_peered":"2023-10-14T11:59:01.178136+0000","last_unstale":"2023-10-14T11:59:02.137138+0000","last_undegraded":"2023-10-14T11:59:02.137138+0000","last_fullsized":"2023-10-14T11:59:02.137138+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:43:52.203439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.139738+0000","last_change":"2023-10-14T11:59:02.139738+0000","last_active":"2023-10-14T11:59:02.139738+0000","last_peered":"2023-10-14T11:59:02.139738+0000","last_clean":"2023-10-14T11:59:02.139738+0000","last_became_active":"2023-10-14T11:59:02.139431+0000","last_became_peered":"2023-10-14T11:59:02.139431+0000","last_unstale":"2023-10-14T11:59:02.139738+0000","last_undegraded":"2023-10-14T11:59:02.139738+0000","last_fullsized":"2023-10-14T11:59:02.139738+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:41:47.404184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.133347+0000","last_change":"2023-10-14T11:59:01.179865+0000","last_active":"2023-10-14T11:59:02.133347+0000","last_peered":"2023-10-14T11:59:02.133347+0000","last_clean":"2023-10-14T11:59:02.133347+0000","last_became_active":"2023-10-14T11:59:01.178404+0000","last_became_peered":"2023-10-14T11:59:01.178404+0000","last_unstale":"2023-10-14T11:59:02.133347+0000","last_undegraded":"2023-10-14T11:59:02.133347+0000","last_fullsized":"2023-10-14T11:59:02.133347+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:30:40.963797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.146195+0000","last_change":"2023-10-14T11:59:02.146195+0000","last_active":"2023-10-14T11:59:02.146195+0000","last_peered":"2023-10-14T11:59:02.146195+0000","last_clean":"2023-10-14T11:59:02.146195+0000","last_became_active":"2023-10-14T11:59:02.144184+0000","last_became_peered":"2023-10-14T11:59:02.144184+0000","last_unstale":"2023-10-14T11:59:02.146195+0000","last_undegraded":"2023-10-14T11:59:02.146195+0000","last_fullsized":"2023-10-14T11:59:02.146195+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:43:53.205090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.135778+0000","last_change":"2023-10-14T11:59:01.185486+0000","last_active":"2023-10-14T11:59:02.135778+0000","last_peered":"2023-10-14T11:59:02.135778+0000","last_clean":"2023-10-14T11:59:02.135778+0000","last_became_active":"2023-10-14T11:59:01.185200+0000","last_became_peered":"2023-10-14T11:59:01.185200+0000","last_unstale":"2023-10-14T11:59:02.135778+0000","last_undegraded":"2023-10-14T11:59:02.135778+0000","last_fullsized":"2023-10-14T11:59:02.135778+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:48:36.787554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.139052+0000","last_change":"2023-10-14T11:59:02.139052+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.139052+0000","last_undegraded":"2023-10-14T11:59:02.139052+0000","last_fullsized":"2023-10-14T11:59:02.139052+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:36:52.548523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.132966+0000","last_change":"2023-10-14T11:59:01.178774+0000","last_active":"2023-10-14T11:59:02.132966+0000","last_peered":"2023-10-14T11:59:02.132966+0000","last_clean":"2023-10-14T11:59:02.132966+0000","last_became_active":"2023-10-14T11:59:01.178378+0000","last_became_peered":"2023-10-14T11:59:01.178378+0000","last_unstale":"2023-10-14T11:59:02.132966+0000","last_undegraded":"2023-10-14T11:59:02.132966+0000","last_fullsized":"2023-10-14T11:59:02.132966+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:25.623744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.149841+0000","last_change":"2023-10-14T11:59:02.143614+0000","last_active":"2023-10-14T11:59:02.149841+0000","last_peered":"2023-10-14T11:59:02.149841+0000","last_clean":"2023-10-14T11:59:02.149841+0000","last_became_active":"2023-10-14T11:59:02.143332+0000","last_became_peered":"2023-10-14T11:59:02.143332+0000","last_unstale":"2023-10-14T11:59:02.149841+0000","last_undegraded":"2023-10-14T11:59:02.149841+0000","last_fullsized":"2023-10-14T11:59:02.149841+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:12:09.205507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.133277+0000","last_change":"2023-10-14T11:59:01.178778+0000","last_active":"2023-10-14T11:59:02.133277+0000","last_peered":"2023-10-14T11:59:02.133277+0000","last_clean":"2023-10-14T11:59:02.133277+0000","last_became_active":"2023-10-14T11:59:01.178440+0000","last_became_peered":"2023-10-14T11:59:01.178440+0000","last_unstale":"2023-10-14T11:59:02.133277+0000","last_undegraded":"2023-10-14T11:59:02.133277+0000","last_fullsized":"2023-10-14T11:59:02.133277+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:18:16.224262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.139185+0000","last_change":"2023-10-14T11:59:02.139185+0000","last_active":"2023-10-14T11:59:02.139185+0000","last_peered":"2023-10-14T11:59:02.139185+0000","last_clean":"2023-10-14T11:59:02.139185+0000","last_became_active":"2023-10-14T11:59:02.138862+0000","last_became_peered":"2023-10-14T11:59:02.138862+0000","last_unstale":"2023-10-14T11:59:02.139185+0000","last_undegraded":"2023-10-14T11:59:02.139185+0000","last_fullsized":"2023-10-14T11:59:02.139185+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:48:45.563805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.132488+0000","last_change":"2023-10-14T11:59:01.175662+0000","last_active":"2023-10-14T11:59:02.132488+0000","last_peered":"2023-10-14T11:59:02.132488+0000","last_clean":"2023-10-14T11:59:02.132488+0000","last_became_active":"2023-10-14T11:59:01.175405+0000","last_became_peered":"2023-10-14T11:59:01.175405+0000","last_unstale":"2023-10-14T11:59:02.132488+0000","last_undegraded":"2023-10-14T11:59:02.132488+0000","last_fullsized":"2023-10-14T11:59:02.132488+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:57:17.506030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.143559+0000","last_change":"2023-10-14T11:59:02.143559+0000","last_active":"2023-10-14T11:59:02.143559+0000","last_peered":"2023-10-14T11:59:02.143559+0000","last_clean":"2023-10-14T11:59:02.143559+0000","last_became_active":"2023-10-14T11:59:02.143303+0000","last_became_peered":"2023-10-14T11:59:02.143303+0000","last_unstale":"2023-10-14T11:59:02.143559+0000","last_undegraded":"2023-10-14T11:59:02.143559+0000","last_fullsized":"2023-10-14T11:59:02.143559+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:20:02.525878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.132854+0000","last_change":"2023-10-14T11:59:01.167233+0000","last_active":"2023-10-14T11:59:02.132854+0000","last_peered":"2023-10-14T11:59:02.132854+0000","last_clean":"2023-10-14T11:59:02.132854+0000","last_became_active":"2023-10-14T11:59:01.166905+0000","last_became_peered":"2023-10-14T11:59:01.166905+0000","last_unstale":"2023-10-14T11:59:02.132854+0000","last_undegraded":"2023-10-14T11:59:02.132854+0000","last_fullsized":"2023-10-14T11:59:02.132854+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:43:52.203439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.152247+0000","last_change":"2023-10-14T11:59:02.150665+0000","last_active":"2023-10-14T11:59:02.152247+0000","last_peered":"2023-10-14T11:59:02.152247+0000","last_clean":"2023-10-14T11:59:02.152247+0000","last_became_active":"2023-10-14T11:59:02.150384+0000","last_became_peered":"2023-10-14T11:59:02.150384+0000","last_unstale":"2023-10-14T11:59:02.152247+0000","last_undegraded":"2023-10-14T11:59:02.152247+0000","last_fullsized":"2023-10-14T11:59:02.152247+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:07:15.007617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.133851+0000","last_change":"2023-10-14T11:59:01.179314+0000","last_active":"2023-10-14T11:59:02.133851+0000","last_peered":"2023-10-14T11:59:02.133851+0000","last_clean":"2023-10-14T11:59:02.133851+0000","last_became_active":"2023-10-14T11:59:01.179019+0000","last_became_peered":"2023-10-14T11:59:01.179019+0000","last_unstale":"2023-10-14T11:59:02.133851+0000","last_undegraded":"2023-10-14T11:59:02.133851+0000","last_fullsized":"2023-10-14T11:59:02.133851+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:27:06.786668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.137973+0000","last_change":"2023-10-14T11:59:02.137973+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.137973+0000","last_undegraded":"2023-10-14T11:59:02.137973+0000","last_fullsized":"2023-10-14T11:59:02.137973+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:36:52.548523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.137264+0000","last_change":"2023-10-14T11:59:01.177873+0000","last_active":"2023-10-14T11:59:02.137264+0000","last_peered":"2023-10-14T11:59:02.137264+0000","last_clean":"2023-10-14T11:59:02.137264+0000","last_became_active":"2023-10-14T11:59:01.177594+0000","last_became_peered":"2023-10-14T11:59:01.177594+0000","last_unstale":"2023-10-14T11:59:02.137264+0000","last_undegraded":"2023-10-14T11:59:02.137264+0000","last_fullsized":"2023-10-14T11:59:02.137264+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:25.623744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.132187+0000","last_change":"2023-10-14T11:59:02.132187+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.132187+0000","last_undegraded":"2023-10-14T11:59:02.132187+0000","last_fullsized":"2023-10-14T11:59:02.132187+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:47:13.361893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.135909+0000","last_change":"2023-10-14T11:59:01.178143+0000","last_active":"2023-10-14T11:59:02.135909+0000","last_peered":"2023-10-14T11:59:02.135909+0000","last_clean":"2023-10-14T11:59:02.135909+0000","last_became_active":"2023-10-14T11:59:01.177736+0000","last_became_peered":"2023-10-14T11:59:01.177736+0000","last_unstale":"2023-10-14T11:59:02.135909+0000","last_undegraded":"2023-10-14T11:59:02.135909+0000","last_fullsized":"2023-10-14T11:59:02.135909+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:35:58.783531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.150847+0000","last_change":"2023-10-14T11:59:02.145816+0000","last_active":"2023-10-14T11:59:02.150847+0000","last_peered":"2023-10-14T11:59:02.150847+0000","last_clean":"2023-10-14T11:59:02.150847+0000","last_became_active":"2023-10-14T11:59:02.145548+0000","last_became_peered":"2023-10-14T11:59:02.145548+0000","last_unstale":"2023-10-14T11:59:02.150847+0000","last_undegraded":"2023-10-14T11:59:02.150847+0000","last_fullsized":"2023-10-14T11:59:02.150847+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:13:03.743718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.133733+0000","last_change":"2023-10-14T11:59:01.180187+0000","last_active":"2023-10-14T11:59:02.133733+0000","last_peered":"2023-10-14T11:59:02.133733+0000","last_clean":"2023-10-14T11:59:02.133733+0000","last_became_active":"2023-10-14T11:59:01.179909+0000","last_became_peered":"2023-10-14T11:59:01.179909+0000","last_unstale":"2023-10-14T11:59:02.133733+0000","last_undegraded":"2023-10-14T11:59:02.133733+0000","last_fullsized":"2023-10-14T11:59:02.133733+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:58:06.846280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.143696+0000","last_change":"2023-10-14T11:59:02.143696+0000","last_active":"2023-10-14T11:59:02.143696+0000","last_peered":"2023-10-14T11:59:02.143696+0000","last_clean":"2023-10-14T11:59:02.143696+0000","last_became_active":"2023-10-14T11:59:02.143443+0000","last_became_peered":"2023-10-14T11:59:02.143443+0000","last_unstale":"2023-10-14T11:59:02.143696+0000","last_undegraded":"2023-10-14T11:59:02.143696+0000","last_fullsized":"2023-10-14T11:59:02.143696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:47:13.361893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.134332+0000","last_change":"2023-10-14T11:59:01.165901+0000","last_active":"2023-10-14T11:59:02.134332+0000","last_peered":"2023-10-14T11:59:02.134332+0000","last_clean":"2023-10-14T11:59:02.134332+0000","last_became_active":"2023-10-14T11:59:01.165597+0000","last_became_peered":"2023-10-14T11:59:01.165597+0000","last_unstale":"2023-10-14T11:59:02.134332+0000","last_undegraded":"2023-10-14T11:59:02.134332+0000","last_fullsized":"2023-10-14T11:59:02.134332+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:05:37.807130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.141427+0000","last_change":"2023-10-14T11:59:02.141427+0000","last_active":"2023-10-14T11:59:02.141427+0000","last_peered":"2023-10-14T11:59:02.141427+0000","last_clean":"2023-10-14T11:59:02.141427+0000","last_became_active":"2023-10-14T11:59:02.141166+0000","last_became_peered":"2023-10-14T11:59:02.141166+0000","last_unstale":"2023-10-14T11:59:02.141427+0000","last_undegraded":"2023-10-14T11:59:02.141427+0000","last_fullsized":"2023-10-14T11:59:02.141427+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:18:17.225913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.136387+0000","last_change":"2023-10-14T11:59:01.172037+0000","last_active":"2023-10-14T11:59:02.136387+0000","last_peered":"2023-10-14T11:59:02.136387+0000","last_clean":"2023-10-14T11:59:02.136387+0000","last_became_active":"2023-10-14T11:59:01.170760+0000","last_became_peered":"2023-10-14T11:59:01.170760+0000","last_unstale":"2023-10-14T11:59:02.136387+0000","last_undegraded":"2023-10-14T11:59:02.136387+0000","last_fullsized":"2023-10-14T11:59:02.136387+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:57:17.506030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.138925+0000","last_change":"2023-10-14T11:59:02.138925+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.138925+0000","last_undegraded":"2023-10-14T11:59:02.138925+0000","last_fullsized":"2023-10-14T11:59:02.138925+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:12:09.205507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.132642+0000","last_change":"2023-10-14T11:59:01.178960+0000","last_active":"2023-10-14T11:59:02.132642+0000","last_peered":"2023-10-14T11:59:02.132642+0000","last_clean":"2023-10-14T11:59:02.132642+0000","last_became_active":"2023-10-14T11:59:01.178691+0000","last_became_peered":"2023-10-14T11:59:01.178691+0000","last_unstale":"2023-10-14T11:59:02.132642+0000","last_undegraded":"2023-10-14T11:59:02.132642+0000","last_fullsized":"2023-10-14T11:59:02.132642+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:13.679459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.140181+0000","last_change":"2023-10-14T11:59:02.140181+0000","last_active":"2023-10-14T11:59:02.140181+0000","last_peered":"2023-10-14T11:59:02.140181+0000","last_clean":"2023-10-14T11:59:02.140181+0000","last_became_active":"2023-10-14T11:59:02.139382+0000","last_became_peered":"2023-10-14T11:59:02.139382+0000","last_unstale":"2023-10-14T11:59:02.140181+0000","last_undegraded":"2023-10-14T11:59:02.140181+0000","last_fullsized":"2023-10-14T11:59:02.140181+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:20:02.525878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.138336+0000","last_change":"2023-10-14T11:59:01.178769+0000","last_active":"2023-10-14T11:59:02.138336+0000","last_peered":"2023-10-14T11:59:02.138336+0000","last_clean":"2023-10-14T11:59:02.138336+0000","last_became_active":"2023-10-14T11:59:01.178288+0000","last_became_peered":"2023-10-14T11:59:01.178288+0000","last_unstale":"2023-10-14T11:59:02.138336+0000","last_undegraded":"2023-10-14T11:59:02.138336+0000","last_fullsized":"2023-10-14T11:59:02.138336+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:07:14.005966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.132368+0000","last_change":"2023-10-14T11:59:02.132368+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.132368+0000","last_undegraded":"2023-10-14T11:59:02.132368+0000","last_fullsized":"2023-10-14T11:59:02.132368+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:05:58.896548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.134614+0000","last_change":"2023-10-14T11:59:01.183630+0000","last_active":"2023-10-14T11:59:02.134614+0000","last_peered":"2023-10-14T11:59:02.134614+0000","last_clean":"2023-10-14T11:59:02.134614+0000","last_became_active":"2023-10-14T11:59:01.183408+0000","last_became_peered":"2023-10-14T11:59:01.183408+0000","last_unstale":"2023-10-14T11:59:02.134614+0000","last_undegraded":"2023-10-14T11:59:02.134614+0000","last_fullsized":"2023-10-14T11:59:02.134614+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:25.623744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.145545+0000","last_change":"2023-10-14T11:59:02.141878+0000","last_active":"2023-10-14T11:59:02.145545+0000","last_peered":"2023-10-14T11:59:02.145545+0000","last_clean":"2023-10-14T11:59:02.145545+0000","last_became_active":"2023-10-14T11:59:02.141229+0000","last_became_peered":"2023-10-14T11:59:02.141229+0000","last_unstale":"2023-10-14T11:59:02.145545+0000","last_undegraded":"2023-10-14T11:59:02.145545+0000","last_fullsized":"2023-10-14T11:59:02.145545+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:11:18.803472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.132715+0000","last_change":"2023-10-14T11:59:01.165942+0000","last_active":"2023-10-14T11:59:02.132715+0000","last_peered":"2023-10-14T11:59:02.132715+0000","last_clean":"2023-10-14T11:59:02.132715+0000","last_became_active":"2023-10-14T11:59:01.165646+0000","last_became_peered":"2023-10-14T11:59:01.165646+0000","last_unstale":"2023-10-14T11:59:02.132715+0000","last_undegraded":"2023-10-14T11:59:02.132715+0000","last_fullsized":"2023-10-14T11:59:02.132715+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:14:59.591440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.147269+0000","last_change":"2023-10-14T11:59:02.147269+0000","last_active":"2023-10-14T11:59:02.147269+0000","last_peered":"2023-10-14T11:59:02.147269+0000","last_clean":"2023-10-14T11:59:02.147269+0000","last_became_active":"2023-10-14T11:59:02.147005+0000","last_became_peered":"2023-10-14T11:59:02.147005+0000","last_unstale":"2023-10-14T11:59:02.147269+0000","last_undegraded":"2023-10-14T11:59:02.147269+0000","last_fullsized":"2023-10-14T11:59:02.147269+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:18:17.225913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.131810+0000","last_change":"2023-10-14T11:59:01.183080+0000","last_active":"2023-10-14T11:59:02.131810+0000","last_peered":"2023-10-14T11:59:02.131810+0000","last_clean":"2023-10-14T11:59:02.131810+0000","last_became_active":"2023-10-14T11:59:01.182786+0000","last_became_peered":"2023-10-14T11:59:01.182786+0000","last_unstale":"2023-10-14T11:59:02.131810+0000","last_undegraded":"2023-10-14T11:59:02.131810+0000","last_fullsized":"2023-10-14T11:59:02.131810+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:05:27.508197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.146618+0000","last_change":"2023-10-14T11:59:02.146618+0000","last_active":"2023-10-14T11:59:02.146618+0000","last_peered":"2023-10-14T11:59:02.146618+0000","last_clean":"2023-10-14T11:59:02.146618+0000","last_became_active":"2023-10-14T11:59:02.146352+0000","last_became_peered":"2023-10-14T11:59:02.146352+0000","last_unstale":"2023-10-14T11:59:02.146618+0000","last_undegraded":"2023-10-14T11:59:02.146618+0000","last_fullsized":"2023-10-14T11:59:02.146618+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:48:45.563805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.132063+0000","last_change":"2023-10-14T11:59:01.178771+0000","last_active":"2023-10-14T11:59:02.132063+0000","last_peered":"2023-10-14T11:59:02.132063+0000","last_clean":"2023-10-14T11:59:02.132063+0000","last_became_active":"2023-10-14T11:59:01.178416+0000","last_became_peered":"2023-10-14T11:59:01.178416+0000","last_unstale":"2023-10-14T11:59:02.132063+0000","last_undegraded":"2023-10-14T11:59:02.132063+0000","last_fullsized":"2023-10-14T11:59:02.132063+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:11:46.659890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.145624+0000","last_change":"2023-10-14T11:59:02.145624+0000","last_active":"2023-10-14T11:59:02.145624+0000","last_peered":"2023-10-14T11:59:02.145624+0000","last_clean":"2023-10-14T11:59:02.145624+0000","last_became_active":"2023-10-14T11:59:02.145381+0000","last_became_peered":"2023-10-14T11:59:02.145381+0000","last_unstale":"2023-10-14T11:59:02.145624+0000","last_undegraded":"2023-10-14T11:59:02.145624+0000","last_fullsized":"2023-10-14T11:59:02.145624+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:21:57.196717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.137214+0000","last_change":"2023-10-14T11:59:01.177556+0000","last_active":"2023-10-14T11:59:02.137214+0000","last_peered":"2023-10-14T11:59:02.137214+0000","last_clean":"2023-10-14T11:59:02.137214+0000","last_became_active":"2023-10-14T11:59:01.177313+0000","last_became_peered":"2023-10-14T11:59:01.177313+0000","last_unstale":"2023-10-14T11:59:02.137214+0000","last_undegraded":"2023-10-14T11:59:02.137214+0000","last_fullsized":"2023-10-14T11:59:02.137214+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:27:02.969220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.141409+0000","last_change":"2023-10-14T11:59:02.141409+0000","last_active":"2023-10-14T11:59:02.141409+0000","last_peered":"2023-10-14T11:59:02.141409+0000","last_clean":"2023-10-14T11:59:02.141409+0000","last_became_active":"2023-10-14T11:59:02.141163+0000","last_became_peered":"2023-10-14T11:59:02.141163+0000","last_unstale":"2023-10-14T11:59:02.141409+0000","last_undegraded":"2023-10-14T11:59:02.141409+0000","last_fullsized":"2023-10-14T11:59:02.141409+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:54:06.957363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.133688+0000","last_change":"2023-10-14T11:59:01.168240+0000","last_active":"2023-10-14T11:59:02.133688+0000","last_peered":"2023-10-14T11:59:02.133688+0000","last_clean":"2023-10-14T11:59:02.133688+0000","last_became_active":"2023-10-14T11:59:01.167944+0000","last_became_peered":"2023-10-14T11:59:01.167944+0000","last_unstale":"2023-10-14T11:59:02.133688+0000","last_undegraded":"2023-10-14T11:59:02.133688+0000","last_fullsized":"2023-10-14T11:59:02.133688+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:27:02.969220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":4,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.140710+0000","last_change":"2023-10-14T11:59:02.133062+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.140710+0000","last_undegraded":"2023-10-14T11:59:02.140710+0000","last_fullsized":"2023-10-14T11:59:02.140710+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:40:14.734059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.133269+0000","last_change":"2023-10-14T11:59:01.176530+0000","last_active":"2023-10-14T11:59:02.133269+0000","last_peered":"2023-10-14T11:59:02.133269+0000","last_clean":"2023-10-14T11:59:02.133269+0000","last_became_active":"2023-10-14T11:59:01.176193+0000","last_became_peered":"2023-10-14T11:59:01.176193+0000","last_unstale":"2023-10-14T11:59:02.133269+0000","last_undegraded":"2023-10-14T11:59:02.133269+0000","last_fullsized":"2023-10-14T11:59:02.133269+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:54:27.847599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.136372+0000","last_change":"2023-10-14T11:59:02.136372+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.136372+0000","last_undegraded":"2023-10-14T11:59:02.136372+0000","last_fullsized":"2023-10-14T11:59:02.136372+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:14.681110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.130535+0000","last_change":"2023-10-14T11:59:01.178157+0000","last_active":"2023-10-14T11:59:02.130535+0000","last_peered":"2023-10-14T11:59:02.130535+0000","last_clean":"2023-10-14T11:59:02.130535+0000","last_became_active":"2023-10-14T11:59:01.177863+0000","last_became_peered":"2023-10-14T11:59:01.177863+0000","last_unstale":"2023-10-14T11:59:02.130535+0000","last_undegraded":"2023-10-14T11:59:02.130535+0000","last_fullsized":"2023-10-14T11:59:02.130535+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:57:17.506030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.143662+0000","last_change":"2023-10-14T11:59:02.143662+0000","last_active":"2023-10-14T11:59:02.143662+0000","last_peered":"2023-10-14T11:59:02.143662+0000","last_clean":"2023-10-14T11:59:02.143662+0000","last_became_active":"2023-10-14T11:59:02.143456+0000","last_became_peered":"2023-10-14T11:59:02.143456+0000","last_unstale":"2023-10-14T11:59:02.143662+0000","last_undegraded":"2023-10-14T11:59:02.143662+0000","last_fullsized":"2023-10-14T11:59:02.143662+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:33:27.410202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.134465+0000","last_change":"2023-10-14T11:59:01.181546+0000","last_active":"2023-10-14T11:59:02.134465+0000","last_peered":"2023-10-14T11:59:02.134465+0000","last_clean":"2023-10-14T11:59:02.134465+0000","last_became_active":"2023-10-14T11:59:01.181255+0000","last_became_peered":"2023-10-14T11:59:01.181255+0000","last_unstale":"2023-10-14T11:59:02.134465+0000","last_undegraded":"2023-10-14T11:59:02.134465+0000","last_fullsized":"2023-10-14T11:59:02.134465+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:27:02.969220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.145565+0000","last_change":"2023-10-14T11:59:02.142119+0000","last_active":"2023-10-14T11:59:02.145565+0000","last_peered":"2023-10-14T11:59:02.145565+0000","last_clean":"2023-10-14T11:59:02.145565+0000","last_became_active":"2023-10-14T11:59:02.141586+0000","last_became_peered":"2023-10-14T11:59:02.141586+0000","last_unstale":"2023-10-14T11:59:02.145565+0000","last_undegraded":"2023-10-14T11:59:02.145565+0000","last_fullsized":"2023-10-14T11:59:02.145565+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:36:52.548523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.137387+0000","last_change":"2023-10-14T11:59:01.180897+0000","last_active":"2023-10-14T11:59:02.137387+0000","last_peered":"2023-10-14T11:59:02.137387+0000","last_clean":"2023-10-14T11:59:02.137387+0000","last_became_active":"2023-10-14T11:59:01.180524+0000","last_became_peered":"2023-10-14T11:59:01.180524+0000","last_unstale":"2023-10-14T11:59:02.137387+0000","last_undegraded":"2023-10-14T11:59:02.137387+0000","last_fullsized":"2023-10-14T11:59:02.137387+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:18:16.224262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.136894+0000","last_change":"2023-10-14T11:59:02.136894+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.136894+0000","last_undegraded":"2023-10-14T11:59:02.136894+0000","last_fullsized":"2023-10-14T11:59:02.136894+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:15:00.593091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.136362+0000","last_change":"2023-10-14T11:59:01.183591+0000","last_active":"2023-10-14T11:59:02.136362+0000","last_peered":"2023-10-14T11:59:02.136362+0000","last_clean":"2023-10-14T11:59:02.136362+0000","last_became_active":"2023-10-14T11:59:01.183019+0000","last_became_peered":"2023-10-14T11:59:01.183019+0000","last_unstale":"2023-10-14T11:59:02.136362+0000","last_undegraded":"2023-10-14T11:59:02.136362+0000","last_fullsized":"2023-10-14T11:59:02.136362+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:05:37.807130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.133356+0000","last_change":"2023-10-14T11:59:02.133356+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.133356+0000","last_undegraded":"2023-10-14T11:59:02.133356+0000","last_fullsized":"2023-10-14T11:59:02.133356+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:39:49.443148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.137080+0000","last_change":"2023-10-14T11:59:01.177539+0000","last_active":"2023-10-14T11:59:02.137080+0000","last_peered":"2023-10-14T11:59:02.137080+0000","last_clean":"2023-10-14T11:59:02.137080+0000","last_became_active":"2023-10-14T11:59:01.177255+0000","last_became_peered":"2023-10-14T11:59:01.177255+0000","last_unstale":"2023-10-14T11:59:02.137080+0000","last_undegraded":"2023-10-14T11:59:02.137080+0000","last_fullsized":"2023-10-14T11:59:02.137080+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:39:48.441497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.137750+0000","last_change":"2023-10-14T11:59:02.137750+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.137750+0000","last_undegraded":"2023-10-14T11:59:02.137750+0000","last_fullsized":"2023-10-14T11:59:02.137750+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:47:13.361893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.136562+0000","last_change":"2023-10-14T11:59:01.182837+0000","last_active":"2023-10-14T11:59:02.136562+0000","last_peered":"2023-10-14T11:59:02.136562+0000","last_clean":"2023-10-14T11:59:02.136562+0000","last_became_active":"2023-10-14T11:59:01.182236+0000","last_became_peered":"2023-10-14T11:59:01.182236+0000","last_unstale":"2023-10-14T11:59:02.136562+0000","last_undegraded":"2023-10-14T11:59:02.136562+0000","last_fullsized":"2023-10-14T11:59:02.136562+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:13.679459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":4,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.138516+0000","last_change":"2023-10-14T11:59:02.138103+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.138516+0000","last_undegraded":"2023-10-14T11:59:02.138516+0000","last_fullsized":"2023-10-14T11:59:02.138516+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:13:03.743718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.133395+0000","last_change":"2023-10-14T11:59:01.168077+0000","last_active":"2023-10-14T11:59:02.133395+0000","last_peered":"2023-10-14T11:59:02.133395+0000","last_clean":"2023-10-14T11:59:02.133395+0000","last_became_active":"2023-10-14T11:59:01.167803+0000","last_became_peered":"2023-10-14T11:59:01.167803+0000","last_unstale":"2023-10-14T11:59:02.133395+0000","last_undegraded":"2023-10-14T11:59:02.133395+0000","last_fullsized":"2023-10-14T11:59:02.133395+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:47:12.360242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.139387+0000","last_change":"2023-10-14T11:59:02.139387+0000","last_active":"2023-10-14T11:59:02.139387+0000","last_peered":"2023-10-14T11:59:02.139387+0000","last_clean":"2023-10-14T11:59:02.139387+0000","last_became_active":"2023-10-14T11:59:02.139117+0000","last_became_peered":"2023-10-14T11:59:02.139117+0000","last_unstale":"2023-10-14T11:59:02.139387+0000","last_undegraded":"2023-10-14T11:59:02.139387+0000","last_fullsized":"2023-10-14T11:59:02.139387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:27:03.970871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.138469+0000","last_change":"2023-10-14T11:59:01.179436+0000","last_active":"2023-10-14T11:59:02.138469+0000","last_peered":"2023-10-14T11:59:02.138469+0000","last_clean":"2023-10-14T11:59:02.138469+0000","last_became_active":"2023-10-14T11:59:01.179109+0000","last_became_peered":"2023-10-14T11:59:01.179109+0000","last_unstale":"2023-10-14T11:59:02.138469+0000","last_undegraded":"2023-10-14T11:59:02.138469+0000","last_fullsized":"2023-10-14T11:59:02.138469+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:11:17.801821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.143298+0000","last_change":"2023-10-14T11:59:02.143298+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.143298+0000","last_undegraded":"2023-10-14T11:59:02.143298+0000","last_fullsized":"2023-10-14T11:59:02.143298+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:13:03.743718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.131329+0000","last_change":"2023-10-14T11:59:01.169175+0000","last_active":"2023-10-14T11:59:02.131329+0000","last_peered":"2023-10-14T11:59:02.131329+0000","last_clean":"2023-10-14T11:59:02.131329+0000","last_became_active":"2023-10-14T11:59:01.168873+0000","last_became_peered":"2023-10-14T11:59:01.168873+0000","last_unstale":"2023-10-14T11:59:02.131329+0000","last_undegraded":"2023-10-14T11:59:02.131329+0000","last_fullsized":"2023-10-14T11:59:02.131329+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:53:43.573010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":172,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-14T11:59:01.174640+0000","last_change":"2023-10-14T11:58:10.616726+0000","last_active":"2023-10-14T11:59:01.174640+0000","last_peered":"2023-10-14T11:59:01.174640+0000","last_clean":"2023-10-14T11:59:01.174640+0000","last_became_active":"2023-10-14T11:58:10.615628+0000","last_became_peered":"2023-10-14T11:58:10.615628+0000","last_unstale":"2023-10-14T11:59:01.174640+0000","last_undegraded":"2023-10-14T11:59:01.174640+0000","last_fullsized":"2023-10-14T11:59:01.174640+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:09.001265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:09.001265+0000","last_clean_scrub_stamp":"2023-10-14T11:58:09.001265+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:22:50.496050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65244,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574862,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":716079,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574862,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":125711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84399999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574862,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":132516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]}]},{"osd":0,"up_from":13,"seq":55834574861,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":401408,"data_stored":716079,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574861,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":125711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.254}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574862,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":125711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574862,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":125711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574862,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":125711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.311}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T11:59:07.046 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3e is not active+clean 2023-10-14T11:59:07.046 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3e', 'version': "0'0", 'reported_seq': 4, 'reported_epoch': 31, 'state': 'creating+activating', 'last_fresh': '2023-10-14T11:59:02.139597+0000', 'last_change': '2023-10-14T11:59:02.138579+0000', 'last_active': '2023-10-14T11:59:01.155233+0000', 'last_peered': '2023-10-14T11:59:01.155233+0000', 'last_clean': '2023-10-14T11:59:01.155233+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T11:59:02.139597+0000', 'last_undegraded': '2023-10-14T11:59:02.139597+0000', 'last_fullsized': '2023-10-14T11:59:02.139597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_clean_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-15T19:35:59.785182+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': ['3', '0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-14T11:59:07.047 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3a is not active+clean 2023-10-14T11:59:07.048 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3a', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 31, 'state': 'creating+activating', 'last_fresh': '2023-10-14T11:59:02.136431+0000', 'last_change': '2023-10-14T11:59:02.136431+0000', 'last_active': '2023-10-14T11:59:01.155233+0000', 'last_peered': '2023-10-14T11:59:01.155233+0000', 'last_clean': '2023-10-14T11:59:01.155233+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T11:59:02.136431+0000', 'last_undegraded': '2023-10-14T11:59:02.136431+0000', 'last_fullsized': '2023-10-14T11:59:02.136431+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_clean_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-15T21:08:25.866669+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': ['3', '5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-14T11:59:07.048 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.39 is not active+clean 2023-10-14T11:59:07.048 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.39', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 31, 'state': 'creating+activating', 'last_fresh': '2023-10-14T11:59:02.135963+0000', 'last_change': '2023-10-14T11:59:02.135963+0000', 'last_active': '2023-10-14T11:59:01.155233+0000', 'last_peered': '2023-10-14T11:59:01.155233+0000', 'last_clean': '2023-10-14T11:59:01.155233+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T11:59:02.135963+0000', 'last_undegraded': '2023-10-14T11:59:02.135963+0000', 'last_fullsized': '2023-10-14T11:59:02.135963+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_clean_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-15T17:05:38.808781+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': ['7', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-14T11:59:07.049 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.38 is not active+clean 2023-10-14T11:59:07.049 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.38', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 31, 'state': 'creating+activating', 'last_fresh': '2023-10-14T11:59:02.135986+0000', 'last_change': '2023-10-14T11:59:02.135986+0000', 'last_active': '2023-10-14T11:59:01.155233+0000', 'last_peered': '2023-10-14T11:59:01.155233+0000', 'last_clean': '2023-10-14T11:59:01.155233+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T11:59:02.135986+0000', 'last_undegraded': '2023-10-14T11:59:02.135986+0000', 'last_fullsized': '2023-10-14T11:59:02.135986+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_clean_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-15T17:57:18.507681+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': ['7', '2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-14T11:59:07.050 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.37 is not active+clean 2023-10-14T11:59:07.050 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.37', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 31, 'state': 'creating+activating', 'last_fresh': '2023-10-14T11:59:02.132676+0000', 'last_change': '2023-10-14T11:59:02.132676+0000', 'last_active': '2023-10-14T11:59:01.155233+0000', 'last_peered': '2023-10-14T11:59:01.155233+0000', 'last_clean': '2023-10-14T11:59:01.155233+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T11:59:02.132676+0000', 'last_undegraded': '2023-10-14T11:59:02.132676+0000', 'last_fullsized': '2023-10-14T11:59:02.132676+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_clean_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-15T14:20:02.525878+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': ['4', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-14T11:59:07.050 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.36 is not active+clean 2023-10-14T11:59:07.051 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.36', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 31, 'state': 'creating+activating', 'last_fresh': '2023-10-14T11:59:02.133724+0000', 'last_change': '2023-10-14T11:59:02.133724+0000', 'last_active': '2023-10-14T11:59:01.155233+0000', 'last_peered': '2023-10-14T11:59:01.155233+0000', 'last_clean': '2023-10-14T11:59:01.155233+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T11:59:02.133724+0000', 'last_undegraded': '2023-10-14T11:59:02.133724+0000', 'last_fullsized': '2023-10-14T11:59:02.133724+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_clean_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-15T23:15:00.593091+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-14T11:59:07.051 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.35 is not active+clean 2023-10-14T11:59:07.052 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.35', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 31, 'state': 'creating+activating', 'last_fresh': '2023-10-14T11:59:02.143474+0000', 'last_change': '2023-10-14T11:59:02.143474+0000', 'last_active': '2023-10-14T11:59:01.155233+0000', 'last_peered': '2023-10-14T11:59:01.155233+0000', 'last_clean': '2023-10-14T11:59:01.155233+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T11:59:02.143474+0000', 'last_undegraded': '2023-10-14T11:59:02.143474+0000', 'last_fullsized': '2023-10-14T11:59:02.143474+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_clean_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-15T22:07:15.007617+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': ['7', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-14T11:59:07.052 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.32 is not active+clean 2023-10-14T11:59:07.052 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.32', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 31, 'state': 'creating+activating', 'last_fresh': '2023-10-14T11:59:02.137158+0000', 'last_change': '2023-10-14T11:59:02.137158+0000', 'last_active': '2023-10-14T11:59:01.155233+0000', 'last_peered': '2023-10-14T11:59:01.155233+0000', 'last_clean': '2023-10-14T11:59:01.155233+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T11:59:02.137158+0000', 'last_undegraded': '2023-10-14T11:59:02.137158+0000', 'last_fullsized': '2023-10-14T11:59:02.137158+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_clean_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-15T23:12:09.205507+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['3', '2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-14T11:59:07.053 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.30 is not active+clean 2023-10-14T11:59:07.053 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.30', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 31, 'state': 'creating+activating', 'last_fresh': '2023-10-14T11:59:02.138148+0000', 'last_change': '2023-10-14T11:59:02.138148+0000', 'last_active': '2023-10-14T11:59:01.155233+0000', 'last_peered': '2023-10-14T11:59:01.155233+0000', 'last_clean': '2023-10-14T11:59:01.155233+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T11:59:02.138148+0000', 'last_undegraded': '2023-10-14T11:59:02.138148+0000', 'last_fullsized': '2023-10-14T11:59:02.138148+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_clean_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-15T13:27:07.788319+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': ['7', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-14T11:59:07.054 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2f is not active+clean 2023-10-14T11:59:07.054 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2f', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 31, 'state': 'creating+activating', 'last_fresh': '2023-10-14T11:59:02.136715+0000', 'last_change': '2023-10-14T11:59:02.136715+0000', 'last_active': '2023-10-14T11:59:01.155233+0000', 'last_peered': '2023-10-14T11:59:01.155233+0000', 'last_clean': '2023-10-14T11:59:01.155233+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T11:59:02.136715+0000', 'last_undegraded': '2023-10-14T11:59:02.136715+0000', 'last_fullsized': '2023-10-14T11:59:02.136715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_clean_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-15T15:30:41.965448+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': ['7', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-14T11:59:07.054 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2e is not active+clean 2023-10-14T11:59:07.055 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2e', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 31, 'state': 'creating+activating', 'last_fresh': '2023-10-14T11:59:02.135337+0000', 'last_change': '2023-10-14T11:59:02.135337+0000', 'last_active': '2023-10-14T11:59:01.155233+0000', 'last_peered': '2023-10-14T11:59:01.155233+0000', 'last_clean': '2023-10-14T11:59:01.155233+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T11:59:02.135337+0000', 'last_undegraded': '2023-10-14T11:59:02.135337+0000', 'last_fullsized': '2023-10-14T11:59:02.135337+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_clean_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-15T17:05:38.808781+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': ['3', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-14T11:59:07.055 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2b is not active+clean 2023-10-14T11:59:07.056 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2b', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 31, 'state': 'creating+activating', 'last_fresh': '2023-10-14T11:59:02.132115+0000', 'last_change': '2023-10-14T11:59:02.132115+0000', 'last_active': '2023-10-14T11:59:01.155233+0000', 'last_peered': '2023-10-14T11:59:01.155233+0000', 'last_clean': '2023-10-14T11:59:01.155233+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T11:59:02.132115+0000', 'last_undegraded': '2023-10-14T11:59:02.132115+0000', 'last_fullsized': '2023-10-14T11:59:02.132115+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_clean_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-15T12:53:44.574661+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': ['4', '0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-14T11:59:07.056 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2a is not active+clean 2023-10-14T11:59:07.057 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2a', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 31, 'state': 'creating+activating', 'last_fresh': '2023-10-14T11:59:02.132815+0000', 'last_change': '2023-10-14T11:59:02.132815+0000', 'last_active': '2023-10-14T11:59:01.155233+0000', 'last_peered': '2023-10-14T11:59:01.155233+0000', 'last_clean': '2023-10-14T11:59:01.155233+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T11:59:02.132815+0000', 'last_undegraded': '2023-10-14T11:59:02.132815+0000', 'last_fullsized': '2023-10-14T11:59:02.132815+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_clean_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-15T20:39:49.443148+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': ['4', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-14T11:59:07.057 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.29 is not active+clean 2023-10-14T11:59:07.058 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.29', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 31, 'state': 'creating+activating', 'last_fresh': '2023-10-14T11:59:02.137732+0000', 'last_change': '2023-10-14T11:59:02.137732+0000', 'last_active': '2023-10-14T11:59:01.155233+0000', 'last_peered': '2023-10-14T11:59:01.155233+0000', 'last_clean': '2023-10-14T11:59:01.155233+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T11:59:02.137732+0000', 'last_undegraded': '2023-10-14T11:59:02.137732+0000', 'last_fullsized': '2023-10-14T11:59:02.137732+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_clean_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-15T21:54:28.849250+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': ['0', '7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-10-14T11:59:07.058 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.26 is not active+clean 2023-10-14T11:59:07.058 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.26', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 31, 'state': 'creating+activating', 'last_fresh': '2023-10-14T11:59:02.132003+0000', 'last_change': '2023-10-14T11:59:02.132003+0000', 'last_active': '2023-10-14T11:59:01.155233+0000', 'last_peered': '2023-10-14T11:59:01.155233+0000', 'last_clean': '2023-10-14T11:59:01.155233+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T11:59:02.132003+0000', 'last_undegraded': '2023-10-14T11:59:02.132003+0000', 'last_fullsized': '2023-10-14T11:59:02.132003+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_clean_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-15T21:54:28.849250+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': ['3', '7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-14T11:59:07.059 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.22 is not active+clean 2023-10-14T11:59:07.059 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.22', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 31, 'state': 'creating+activating', 'last_fresh': '2023-10-14T11:59:02.136790+0000', 'last_change': '2023-10-14T11:59:02.136790+0000', 'last_active': '2023-10-14T11:59:01.155233+0000', 'last_peered': '2023-10-14T11:59:01.155233+0000', 'last_clean': '2023-10-14T11:59:01.155233+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T11:59:02.136790+0000', 'last_undegraded': '2023-10-14T11:59:02.136790+0000', 'last_fullsized': '2023-10-14T11:59:02.136790+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_clean_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-15T15:58:07.847931+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': ['7', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-14T11:59:07.060 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.21 is not active+clean 2023-10-14T11:59:07.060 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.21', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 31, 'state': 'creating+activating', 'last_fresh': '2023-10-14T11:59:02.131455+0000', 'last_change': '2023-10-14T11:59:02.131455+0000', 'last_active': '2023-10-14T11:59:01.155233+0000', 'last_peered': '2023-10-14T11:59:01.155233+0000', 'last_clean': '2023-10-14T11:59:01.155233+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T11:59:02.131455+0000', 'last_undegraded': '2023-10-14T11:59:02.131455+0000', 'last_fullsized': '2023-10-14T11:59:02.131455+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_clean_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-15T13:05:58.896548+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': ['6', '2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-10-14T11:59:07.060 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.20 is not active+clean 2023-10-14T11:59:07.061 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.20', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 31, 'state': 'creating+activating', 'last_fresh': '2023-10-14T11:59:02.138555+0000', 'last_change': '2023-10-14T11:59:02.138555+0000', 'last_active': '2023-10-14T11:59:01.155233+0000', 'last_peered': '2023-10-14T11:59:01.155233+0000', 'last_clean': '2023-10-14T11:59:01.155233+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T11:59:02.138555+0000', 'last_undegraded': '2023-10-14T11:59:02.138555+0000', 'last_fullsized': '2023-10-14T11:59:02.138555+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_clean_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-15T21:08:25.866669+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': ['7', '4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-14T11:59:07.061 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1e is not active+clean 2023-10-14T11:59:07.062 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1e', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 31, 'state': 'creating+activating', 'last_fresh': '2023-10-14T11:59:02.136873+0000', 'last_change': '2023-10-14T11:59:02.136873+0000', 'last_active': '2023-10-14T11:59:01.155233+0000', 'last_peered': '2023-10-14T11:59:01.155233+0000', 'last_clean': '2023-10-14T11:59:01.155233+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T11:59:02.136873+0000', 'last_undegraded': '2023-10-14T11:59:02.136873+0000', 'last_fullsized': '2023-10-14T11:59:02.136873+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_clean_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-15T14:20:02.525878+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': ['0', '7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-10-14T11:59:07.062 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1a is not active+clean 2023-10-14T11:59:07.062 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1a', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 31, 'state': 'creating+activating', 'last_fresh': '2023-10-14T11:59:02.139052+0000', 'last_change': '2023-10-14T11:59:02.139052+0000', 'last_active': '2023-10-14T11:59:01.155233+0000', 'last_peered': '2023-10-14T11:59:01.155233+0000', 'last_clean': '2023-10-14T11:59:01.155233+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T11:59:02.139052+0000', 'last_undegraded': '2023-10-14T11:59:02.139052+0000', 'last_fullsized': '2023-10-14T11:59:02.139052+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_clean_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-15T14:36:52.548523+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': ['7', '4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-14T11:59:07.063 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.17 is not active+clean 2023-10-14T11:59:07.063 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.17', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 31, 'state': 'creating+activating', 'last_fresh': '2023-10-14T11:59:02.137973+0000', 'last_change': '2023-10-14T11:59:02.137973+0000', 'last_active': '2023-10-14T11:59:01.155233+0000', 'last_peered': '2023-10-14T11:59:01.155233+0000', 'last_clean': '2023-10-14T11:59:01.155233+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T11:59:02.137973+0000', 'last_undegraded': '2023-10-14T11:59:02.137973+0000', 'last_fullsized': '2023-10-14T11:59:02.137973+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_clean_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-15T14:36:52.548523+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': ['3', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-14T11:59:07.064 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.14 is not active+clean 2023-10-14T11:59:07.064 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.14', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 31, 'state': 'creating+activating', 'last_fresh': '2023-10-14T11:59:02.132187+0000', 'last_change': '2023-10-14T11:59:02.132187+0000', 'last_active': '2023-10-14T11:59:01.155233+0000', 'last_peered': '2023-10-14T11:59:01.155233+0000', 'last_clean': '2023-10-14T11:59:01.155233+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T11:59:02.132187+0000', 'last_undegraded': '2023-10-14T11:59:02.132187+0000', 'last_fullsized': '2023-10-14T11:59:02.132187+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_clean_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-15T14:47:13.361893+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': ['3', '7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-14T11:59:07.064 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.10 is not active+clean 2023-10-14T11:59:07.065 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.10', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 31, 'state': 'creating+activating', 'last_fresh': '2023-10-14T11:59:02.138925+0000', 'last_change': '2023-10-14T11:59:02.138925+0000', 'last_active': '2023-10-14T11:59:01.155233+0000', 'last_peered': '2023-10-14T11:59:01.155233+0000', 'last_clean': '2023-10-14T11:59:01.155233+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T11:59:02.138925+0000', 'last_undegraded': '2023-10-14T11:59:02.138925+0000', 'last_fullsized': '2023-10-14T11:59:02.138925+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_clean_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-15T23:12:09.205507+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': ['7', '4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-14T11:59:07.065 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.e is not active+clean 2023-10-14T11:59:07.066 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.e', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 31, 'state': 'creating+activating', 'last_fresh': '2023-10-14T11:59:02.132368+0000', 'last_change': '2023-10-14T11:59:02.132368+0000', 'last_active': '2023-10-14T11:59:01.155233+0000', 'last_peered': '2023-10-14T11:59:01.155233+0000', 'last_clean': '2023-10-14T11:59:01.155233+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T11:59:02.132368+0000', 'last_undegraded': '2023-10-14T11:59:02.132368+0000', 'last_fullsized': '2023-10-14T11:59:02.132368+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_clean_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-15T13:05:58.896548+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': ['4', '5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-14T11:59:07.066 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.8 is not active+clean 2023-10-14T11:59:07.066 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.8', 'version': "0'0", 'reported_seq': 4, 'reported_epoch': 31, 'state': 'creating+activating', 'last_fresh': '2023-10-14T11:59:02.140710+0000', 'last_change': '2023-10-14T11:59:02.133062+0000', 'last_active': '2023-10-14T11:59:01.155233+0000', 'last_peered': '2023-10-14T11:59:01.155233+0000', 'last_clean': '2023-10-14T11:59:01.155233+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T11:59:02.140710+0000', 'last_undegraded': '2023-10-14T11:59:02.140710+0000', 'last_fullsized': '2023-10-14T11:59:02.140710+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_clean_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-15T20:40:14.734059+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': ['2', '0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-14T11:59:07.067 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.9 is not active+clean 2023-10-14T11:59:07.067 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.9', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 31, 'state': 'creating+activating', 'last_fresh': '2023-10-14T11:59:02.136372+0000', 'last_change': '2023-10-14T11:59:02.136372+0000', 'last_active': '2023-10-14T11:59:01.155233+0000', 'last_peered': '2023-10-14T11:59:01.155233+0000', 'last_clean': '2023-10-14T11:59:01.155233+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T11:59:02.136372+0000', 'last_undegraded': '2023-10-14T11:59:02.136372+0000', 'last_fullsized': '2023-10-14T11:59:02.136372+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_clean_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-15T19:10:14.681110+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': ['7', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-14T11:59:07.068 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.5 is not active+clean 2023-10-14T11:59:07.068 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.5', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 31, 'state': 'creating+activating', 'last_fresh': '2023-10-14T11:59:02.136894+0000', 'last_change': '2023-10-14T11:59:02.136894+0000', 'last_active': '2023-10-14T11:59:01.155233+0000', 'last_peered': '2023-10-14T11:59:01.155233+0000', 'last_clean': '2023-10-14T11:59:01.155233+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T11:59:02.136894+0000', 'last_undegraded': '2023-10-14T11:59:02.136894+0000', 'last_fullsized': '2023-10-14T11:59:02.136894+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_clean_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-15T23:15:00.593091+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': ['0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-10-14T11:59:07.068 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2 is not active+clean 2023-10-14T11:59:07.069 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 31, 'state': 'creating+activating', 'last_fresh': '2023-10-14T11:59:02.133356+0000', 'last_change': '2023-10-14T11:59:02.133356+0000', 'last_active': '2023-10-14T11:59:01.155233+0000', 'last_peered': '2023-10-14T11:59:01.155233+0000', 'last_clean': '2023-10-14T11:59:01.155233+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T11:59:02.133356+0000', 'last_undegraded': '2023-10-14T11:59:02.133356+0000', 'last_fullsized': '2023-10-14T11:59:02.133356+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_clean_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-15T20:39:49.443148+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': ['6', '0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-10-14T11:59:07.069 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3 is not active+clean 2023-10-14T11:59:07.070 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 31, 'state': 'creating+activating', 'last_fresh': '2023-10-14T11:59:02.137750+0000', 'last_change': '2023-10-14T11:59:02.137750+0000', 'last_active': '2023-10-14T11:59:01.155233+0000', 'last_peered': '2023-10-14T11:59:01.155233+0000', 'last_clean': '2023-10-14T11:59:01.155233+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T11:59:02.137750+0000', 'last_undegraded': '2023-10-14T11:59:02.137750+0000', 'last_fullsized': '2023-10-14T11:59:02.137750+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_clean_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-15T14:47:13.361893+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': ['0', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-10-14T11:59:07.070 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.0 is not active+clean 2023-10-14T11:59:07.070 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.0', 'version': "0'0", 'reported_seq': 4, 'reported_epoch': 31, 'state': 'creating+activating', 'last_fresh': '2023-10-14T11:59:02.138516+0000', 'last_change': '2023-10-14T11:59:02.138103+0000', 'last_active': '2023-10-14T11:59:01.155233+0000', 'last_peered': '2023-10-14T11:59:01.155233+0000', 'last_clean': '2023-10-14T11:59:01.155233+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T11:59:02.138516+0000', 'last_undegraded': '2023-10-14T11:59:02.138516+0000', 'last_fullsized': '2023-10-14T11:59:02.138516+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_clean_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-15T21:13:03.743718+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': ['3', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-14T11:59:07.071 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.4 is not active+clean 2023-10-14T11:59:07.071 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.4', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 31, 'state': 'creating+activating', 'last_fresh': '2023-10-14T11:59:02.143298+0000', 'last_change': '2023-10-14T11:59:02.143298+0000', 'last_active': '2023-10-14T11:59:01.155233+0000', 'last_peered': '2023-10-14T11:59:01.155233+0000', 'last_clean': '2023-10-14T11:59:01.155233+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T11:59:02.143298+0000', 'last_undegraded': '2023-10-14T11:59:02.143298+0000', 'last_fullsized': '2023-10-14T11:59:02.143298+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'last_clean_scrub_stamp': '2023-10-14T11:59:01.155233+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-15T21:13:03.743718+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': ['7', '2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-14T11:59:07.072 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-10-14T11:59:07.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.200+0000 7fa5ff9e7700 1 Processor -- start 2023-10-14T11:59:07.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.200+0000 7fa5ff9e7700 1 -- start start 2023-10-14T11:59:07.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.201+0000 7fa5ff9e7700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa5f806db50 0x7fa5f8072580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:07.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.201+0000 7fa5ff9e7700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5f80c25e0 con 0x7fa5f814b6b0 2023-10-14T11:59:07.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.201+0000 7fa5ff9e7700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5f814ba90 con 0x7fa5f8154bb0 2023-10-14T11:59:07.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.201+0000 7fa5ff9e7700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f80a87b0 con 0x7fa5f806db50 2023-10-14T11:59:07.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.201+0000 7fa5fd783700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fa5f814b6b0 0x7fa5f80c2a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51208/0 (socket says 172.21.15.78:51208) 2023-10-14T11:59:07.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.201+0000 7fa5fd783700 1 -- 172.21.15.78:0/1982154455 learned_addr learned my addr 172.21.15.78:0/1982154455 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:59:07.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.201+0000 7fa5fcf82700 1 --2- 172.21.15.78:0/1982154455 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa5f806db50 0x7fa5f8072580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:07.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.201+0000 7fa5ebfff700 1 -- 172.21.15.78:0/1982154455 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3443249786 0 0) 0x7fa5f80c25e0 con 0x7fa5f814b6b0 2023-10-14T11:59:07.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.201+0000 7fa5ebfff700 1 -- 172.21.15.78:0/1982154455 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5e40035b0 con 0x7fa5f814b6b0 2023-10-14T11:59:07.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.201+0000 7fa5ebfff700 1 -- 172.21.15.78:0/1982154455 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2353369194 0 0) 0x7fa5f814ba90 con 0x7fa5f8154bb0 2023-10-14T11:59:07.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.201+0000 7fa5ebfff700 1 -- 172.21.15.78:0/1982154455 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5f80c25e0 con 0x7fa5f8154bb0 2023-10-14T11:59:07.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.201+0000 7fa5ebfff700 1 -- 172.21.15.78:0/1982154455 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2356902201 0 0) 0x7fa5e40035b0 con 0x7fa5f814b6b0 2023-10-14T11:59:07.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.201+0000 7fa5ebfff700 1 -- 172.21.15.78:0/1982154455 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa5f806db50 msgr2=0x7fa5f8072580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:07.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.201+0000 7fa5ebfff700 1 --2- 172.21.15.78:0/1982154455 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa5f806db50 0x7fa5f8072580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:07.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.201+0000 7fa5ebfff700 1 -- 172.21.15.78:0/1982154455 >> v1:172.21.15.195:6789/0 conn(0x7fa5f8154bb0 legacy=0x7fa5f815bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:07.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.201+0000 7fa5ebfff700 1 -- 172.21.15.78:0/1982154455 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f8155660 con 0x7fa5f814b6b0 2023-10-14T11:59:07.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.201+0000 7fa5ebfff700 1 -- 172.21.15.78:0/1982154455 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441799917 0 0) 0x7fa5f4001670 con 0x7fa5f814b6b0 2023-10-14T11:59:07.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.201+0000 7fa5ebfff700 1 -- 172.21.15.78:0/1982154455 >> v1:172.21.15.78:6789/0 conn(0x7fa5f814b6b0 legacy=0x7fa5f80c2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:07.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.201+0000 7fa5ebfff700 1 --2- 172.21.15.78:0/1982154455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5e4003bc0 0x7fa5e4003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:07.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.201+0000 7fa5ebfff700 1 --2- 172.21.15.78:0/1982154455 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5e4004850 0x7fa5e4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:07.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.201+0000 7fa5ebfff700 1 --2- 172.21.15.78:0/1982154455 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa5f806db50 0x7fa5e4005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:07.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.201+0000 7fa5ebfff700 1 -- 172.21.15.78:0/1982154455 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f80a87b0 con 0x7fa5e4003bc0 2023-10-14T11:59:07.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.201+0000 7fa5ebfff700 1 -- 172.21.15.78:0/1982154455 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e400dd30 con 0x7fa5e4004850 2023-10-14T11:59:07.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.201+0000 7fa5ebfff700 1 -- 172.21.15.78:0/1982154455 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e400def0 con 0x7fa5f806db50 2023-10-14T11:59:07.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.201+0000 7fa5fd783700 1 --2- 172.21.15.78:0/1982154455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5e4003bc0 0x7fa5e4003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:07.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.201+0000 7fa5fcf82700 1 --2- 172.21.15.78:0/1982154455 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5e4004850 0x7fa5e4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:07.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.202+0000 7fa5fdf84700 1 --2- 172.21.15.78:0/1982154455 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa5f806db50 0x7fa5e4005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:07.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.202+0000 7fa5fd783700 1 -- 172.21.15.78:0/1982154455 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa5f806db50 msgr2=0x7fa5e4005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:07.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.202+0000 7fa5fd783700 1 --2- 172.21.15.78:0/1982154455 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa5f806db50 0x7fa5e4005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:07.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.202+0000 7fa5fd783700 1 -- 172.21.15.78:0/1982154455 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5e4004850 msgr2=0x7fa5e4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:07.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.202+0000 7fa5fd783700 1 --2- 172.21.15.78:0/1982154455 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5e4004850 0x7fa5e4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:07.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.202+0000 7fa5fd783700 1 -- 172.21.15.78:0/1982154455 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e40035b0 con 0x7fa5e4003bc0 2023-10-14T11:59:07.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.202+0000 7fa5fd783700 1 --2- 172.21.15.78:0/1982154455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5e4003bc0 0x7fa5e4003fc0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fa5f400c210 tx=0x7fa5e4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ca9e7fafad447ac server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:07.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.202+0000 7fa5ebfff700 1 -- 172.21.15.78:0/1982154455 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f400cdb0 con 0x7fa5e4003bc0 2023-10-14T11:59:07.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.202+0000 7fa5ebfff700 1 -- 172.21.15.78:0/1982154455 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5f4001af0 con 0x7fa5e4003bc0 2023-10-14T11:59:07.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.202+0000 7fa5ebfff700 1 -- 172.21.15.78:0/1982154455 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f4005ba0 con 0x7fa5e4003bc0 2023-10-14T11:59:07.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.203+0000 7fa5ff9e7700 1 -- 172.21.15.78:0/1982154455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5e4003bc0 msgr2=0x7fa5e4003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:07.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.203+0000 7fa5ff9e7700 1 --2- 172.21.15.78:0/1982154455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5e4003bc0 0x7fa5e4003fc0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fa5f400c210 tx=0x7fa5e4005880 comp rx=0 tx=0).stop 2023-10-14T11:59:07.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.203+0000 7fa5fdf84700 1 -- 172.21.15.78:0/1982154455 reap_dead start 2023-10-14T11:59:07.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.203+0000 7fa5ff9e7700 1 -- 172.21.15.78:0/1982154455 shutdown_connections 2023-10-14T11:59:07.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.203+0000 7fa5ff9e7700 1 -- 172.21.15.78:0/1982154455 >> 172.21.15.78:0/1982154455 conn(0x7fa5f80c3ac0 msgr2=0x7fa5f80c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:07.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.203+0000 7fa5ff9e7700 1 -- 172.21.15.78:0/1982154455 shutdown_connections 2023-10-14T11:59:07.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.203+0000 7fa5ff9e7700 1 -- 172.21.15.78:0/1982154455 wait complete. 2023-10-14T11:59:07.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.204+0000 7fa5ff9e7700 1 Processor -- start 2023-10-14T11:59:07.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.204+0000 7fa5ff9e7700 1 -- start start 2023-10-14T11:59:07.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.205+0000 7fa5ff9e7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5e4003bc0 0x7fa5f81418a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:07.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.205+0000 7fa5ff9e7700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa5f80c05c0 0x7fa5f814d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:07.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.205+0000 7fa5ff9e7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5f814d900 0x7fa5f814beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:07.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.205+0000 7fa5ff9e7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f81e7720 con 0x7fa5e4003bc0 2023-10-14T11:59:07.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.205+0000 7fa5ff9e7700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f8143850 con 0x7fa5f814d900 2023-10-14T11:59:07.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.205+0000 7fa5ff9e7700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f814cf30 con 0x7fa5f80c05c0 2023-10-14T11:59:07.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.205+0000 7fa5fd783700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5e4003bc0 0x7fa5f81418a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:07.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.205+0000 7fa5fd783700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5e4003bc0 0x7fa5f81418a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46344/0 (socket says 172.21.15.78:46344) 2023-10-14T11:59:07.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.205+0000 7fa5fd783700 1 -- 172.21.15.78:0/213427344 learned_addr learned my addr 172.21.15.78:0/213427344 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:07.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.205+0000 7fa5fcf82700 1 --2- 172.21.15.78:0/213427344 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa5f80c05c0 0x7fa5f814d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:07.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.205+0000 7fa5fdf84700 1 --2- 172.21.15.78:0/213427344 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5f814d900 0x7fa5f814beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:07.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.205+0000 7fa5fd783700 1 -- 172.21.15.78:0/213427344 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa5f80c05c0 msgr2=0x7fa5f814d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:07.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.205+0000 7fa5fd783700 1 --2- 172.21.15.78:0/213427344 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa5f80c05c0 0x7fa5f814d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:07.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.205+0000 7fa5fd783700 1 -- 172.21.15.78:0/213427344 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5f814d900 msgr2=0x7fa5f814beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:07.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.205+0000 7fa5fd783700 1 --2- 172.21.15.78:0/213427344 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5f814d900 0x7fa5f814beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:07.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.205+0000 7fa5fd783700 1 -- 172.21.15.78:0/213427344 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f8141fa0 con 0x7fa5e4003bc0 2023-10-14T11:59:07.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.205+0000 7fa5fd783700 1 --2- 172.21.15.78:0/213427344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5e4003bc0 0x7fa5f81418a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fa5f400c800 tx=0x7fa5f4001980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:07.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.206+0000 7fa5e9ffb700 1 -- 172.21.15.78:0/213427344 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f40070d0 con 0x7fa5e4003bc0 2023-10-14T11:59:07.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.206+0000 7fa5e9ffb700 1 -- 172.21.15.78:0/213427344 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5f4007230 con 0x7fa5e4003bc0 2023-10-14T11:59:07.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.206+0000 7fa5ff9e7700 1 -- 172.21.15.78:0/213427344 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5f8142280 con 0x7fa5e4003bc0 2023-10-14T11:59:07.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.206+0000 7fa5e9ffb700 1 -- 172.21.15.78:0/213427344 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f40075e0 con 0x7fa5e4003bc0 2023-10-14T11:59:07.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.206+0000 7fa5ff9e7700 1 -- 172.21.15.78:0/213427344 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5f8142840 con 0x7fa5e4003bc0 2023-10-14T11:59:07.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.207+0000 7fa5ff9e7700 1 -- 172.21.15.78:0/213427344 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5f8142a00 con 0x7fa5e4003bc0 2023-10-14T11:59:07.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.209+0000 7fa5e9ffb700 1 -- 172.21.15.78:0/213427344 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa5f4030430 con 0x7fa5e4003bc0 2023-10-14T11:59:07.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.209+0000 7fa5e9ffb700 1 --2- 172.21.15.78:0/213427344 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa5dc06af30 0x7fa5dc06d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:07.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.209+0000 7fa5e9ffb700 1 -- 172.21.15.78:0/213427344 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7fa5f408a830 con 0x7fa5e4003bc0 2023-10-14T11:59:07.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.209+0000 7fa5fcf82700 1 --2- 172.21.15.78:0/213427344 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa5dc06af30 0x7fa5dc06d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:07.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.210+0000 7fa5fcf82700 1 --2- 172.21.15.78:0/213427344 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa5dc06af30 0x7fa5dc06d3f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa5f80ac590 tx=0x7fa5ec009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:07.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.213+0000 7fa5e9ffb700 1 -- 172.21.15.78:0/213427344 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa5f4055a30 con 0x7fa5e4003bc0 2023-10-14T11:59:07.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.485+0000 7fa5ff9e7700 1 -- 172.21.15.78:0/213427344 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fa5f81e7720 con 0x7fa5e4003bc0 2023-10-14T11:59:07.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.485+0000 7fa5e9ffb700 1 -- 172.21.15.78:0/213427344 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+2648 (secure 0 0 0) 0x7fa5f4059050 con 0x7fa5e4003bc0 2023-10-14T11:59:07.488 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:59:07.488 INFO:teuthology.orchestra.run.smithi078.stdout:{"fsid":"4c5dc134-3092-406a-a1e1-13e6d9e926e8","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":70,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":32,"num_osds":8,"num_up_osds":8,"osd_up_since":1697284686,"num_in_osds":8,"osd_in_since":1697284679,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":66},{"state_name":"creating+activating","count":31}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":224186368,"bytes_avail":772869926912,"bytes_total":773094113280,"inactive_pgs_ratio":0.31958761811256409},"fsmap":{"epoch":17,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"d","status":"up:active","gid":5031}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2023-10-14T11:59:04.975482+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"4":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"5":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"6":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"7":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2023-10-14T11:59:07.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.487+0000 7fa5ff9e7700 1 -- 172.21.15.78:0/213427344 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa5dc06af30 msgr2=0x7fa5dc06d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:07.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.487+0000 7fa5ff9e7700 1 --2- 172.21.15.78:0/213427344 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa5dc06af30 0x7fa5dc06d3f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa5f80ac590 tx=0x7fa5ec009210 comp rx=0 tx=0).stop 2023-10-14T11:59:07.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.487+0000 7fa5ff9e7700 1 -- 172.21.15.78:0/213427344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5e4003bc0 msgr2=0x7fa5f81418a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:07.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.487+0000 7fa5ff9e7700 1 --2- 172.21.15.78:0/213427344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5e4003bc0 0x7fa5f81418a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fa5f400c800 tx=0x7fa5f4001980 comp rx=0 tx=0).stop 2023-10-14T11:59:07.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.487+0000 7fa5ff9e7700 1 -- 172.21.15.78:0/213427344 shutdown_connections 2023-10-14T11:59:07.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.487+0000 7fa5ff9e7700 1 --2- 172.21.15.78:0/213427344 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fa5dc06af30 0x7fa5dc06d3f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:07.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.487+0000 7fa5ff9e7700 1 --2- 172.21.15.78:0/213427344 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5f814d900 0x7fa5f814beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:07.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.487+0000 7fa5ff9e7700 1 --2- 172.21.15.78:0/213427344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5e4003bc0 0x7fa5f81418a0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:07.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.487+0000 7fa5ff9e7700 1 --2- 172.21.15.78:0/213427344 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa5f80c05c0 0x7fa5f814d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:07.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.487+0000 7fa5ff9e7700 1 -- 172.21.15.78:0/213427344 >> 172.21.15.78:0/213427344 conn(0x7fa5f80c3ac0 msgr2=0x7fa5f80c2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:07.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.487+0000 7fa5ff9e7700 1 -- 172.21.15.78:0/213427344 shutdown_connections 2023-10-14T11:59:07.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.488+0000 7fa5ff9e7700 1 -- 172.21.15.78:0/213427344 wait complete. 2023-10-14T11:59:07.501 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2023-10-14T11:59:07.502 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T11:59:07.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.621+0000 7ff4aa538700 1 Processor -- start 2023-10-14T11:59:07.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.621+0000 7ff4aa538700 1 -- start start 2023-10-14T11:59:07.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.621+0000 7ff4aa538700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff4a406f1e0 0x7ff4a41554e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:07.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.621+0000 7ff4aa538700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff4a40c0350 0x7ff4a413d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:07.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.621+0000 7ff4aa538700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4a406b630 con 0x7ff4a406ab40 2023-10-14T11:59:07.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.621+0000 7ff4aa538700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4a40c49f0 con 0x7ff4a40c0350 2023-10-14T11:59:07.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.621+0000 7ff4aa538700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4a40726a0 con 0x7ff4a406f1e0 2023-10-14T11:59:07.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.622+0000 7ff4a3fff700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7ff4a406ab40 0x7ff4a406af20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51228/0 (socket says 172.21.15.78:51228) 2023-10-14T11:59:07.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.622+0000 7ff4a3fff700 1 -- 172.21.15.78:0/3886781394 learned_addr learned my addr 172.21.15.78:0/3886781394 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:59:07.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.622+0000 7ff4a8ad5700 1 --2- 172.21.15.78:0/3886781394 >> v2:172.21.15.78:3300/0 conn(0x7ff4a40c0350 0x7ff4a413d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:07.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.622+0000 7ff4a2ffd700 1 -- 172.21.15.78:0/3886781394 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 369107066 0 0) 0x7ff4a406b630 con 0x7ff4a406ab40 2023-10-14T11:59:07.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.622+0000 7ff4a2ffd700 1 -- 172.21.15.78:0/3886781394 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff48c0035b0 con 0x7ff4a406ab40 2023-10-14T11:59:07.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.622+0000 7ff4a37fe700 1 --2- 172.21.15.78:0/3886781394 >> v2:172.21.15.195:3300/0 conn(0x7ff4a406f1e0 0x7ff4a41554e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:07.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.622+0000 7ff4a8ad5700 1 -- 172.21.15.78:0/3886781394 >> v2:172.21.15.195:3300/0 conn(0x7ff4a406f1e0 msgr2=0x7ff4a41554e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:07.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.622+0000 7ff4a8ad5700 1 --2- 172.21.15.78:0/3886781394 >> v2:172.21.15.195:3300/0 conn(0x7ff4a406f1e0 0x7ff4a41554e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:07.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.622+0000 7ff4a8ad5700 1 -- 172.21.15.78:0/3886781394 >> v1:172.21.15.78:6789/0 conn(0x7ff4a406ab40 legacy=0x7ff4a406af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:07.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.622+0000 7ff4a8ad5700 1 -- 172.21.15.78:0/3886781394 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a4072980 con 0x7ff4a40c0350 2023-10-14T11:59:07.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.622+0000 7ff4a8ad5700 1 --2- 172.21.15.78:0/3886781394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4a40c0350 0x7ff4a413d7b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ff4a406ff00 tx=0x7ff49c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abe8b7ee4766b252 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:07.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.622+0000 7ff4a2ffd700 1 -- 172.21.15.78:0/3886781394 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49c00e070 con 0x7ff4a40c0350 2023-10-14T11:59:07.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.622+0000 7ff4a2ffd700 1 -- 172.21.15.78:0/3886781394 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff49c00bb20 con 0x7ff4a40c0350 2023-10-14T11:59:07.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.622+0000 7ff4a2ffd700 1 -- 172.21.15.78:0/3886781394 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49c00be90 con 0x7ff4a40c0350 2023-10-14T11:59:07.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.623+0000 7ff4aa538700 1 -- 172.21.15.78:0/3886781394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4a40c0350 msgr2=0x7ff4a413d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:07.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.623+0000 7ff4aa538700 1 --2- 172.21.15.78:0/3886781394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4a40c0350 0x7ff4a413d7b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ff4a406ff00 tx=0x7ff49c00b330 comp rx=0 tx=0).stop 2023-10-14T11:59:07.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.623+0000 7ff4aa538700 1 -- 172.21.15.78:0/3886781394 shutdown_connections 2023-10-14T11:59:07.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.623+0000 7ff4aa538700 1 --2- 172.21.15.78:0/3886781394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4a40c0350 0x7ff4a413d7b0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:07.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.623+0000 7ff4aa538700 1 --2- 172.21.15.78:0/3886781394 >> v2:172.21.15.195:3300/0 conn(0x7ff4a406f1e0 0x7ff4a41554e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:07.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.623+0000 7ff4aa538700 1 -- 172.21.15.78:0/3886781394 >> 172.21.15.78:0/3886781394 conn(0x7ff4a40c3ac0 msgr2=0x7ff4a40c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:07.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.623+0000 7ff4aa538700 1 -- 172.21.15.78:0/3886781394 shutdown_connections 2023-10-14T11:59:07.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.623+0000 7ff4aa538700 1 -- 172.21.15.78:0/3886781394 wait complete. 2023-10-14T11:59:07.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.624+0000 7ff4aa538700 1 Processor -- start 2023-10-14T11:59:07.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.624+0000 7ff4aa538700 1 -- start start 2023-10-14T11:59:07.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.625+0000 7ff4aa538700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff4a406ab40 0x7ff4a41ea440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:07.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.625+0000 7ff4aa538700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4a406f1e0 0x7ff4a4153a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:07.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.625+0000 7ff4aa538700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff4a40c0350 0x7ff4a41503f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:07.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.625+0000 7ff4aa538700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a4158e10 con 0x7ff4a406f1e0 2023-10-14T11:59:07.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.625+0000 7ff4aa538700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a414f060 con 0x7ff4a40c0350 2023-10-14T11:59:07.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.625+0000 7ff4aa538700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a40c5070 con 0x7ff4a406ab40 2023-10-14T11:59:07.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.625+0000 7ff4a37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4a406f1e0 0x7ff4a4153a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:07.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.625+0000 7ff4a37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4a406f1e0 0x7ff4a4153a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46360/0 (socket says 172.21.15.78:46360) 2023-10-14T11:59:07.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.625+0000 7ff4a37fe700 1 -- 172.21.15.78:0/3157204282 learned_addr learned my addr 172.21.15.78:0/3157204282 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:07.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.625+0000 7ff4a3fff700 1 --2- 172.21.15.78:0/3157204282 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff4a406ab40 0x7ff4a41ea440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:07.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.625+0000 7ff4a8ad5700 1 --2- 172.21.15.78:0/3157204282 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff4a40c0350 0x7ff4a41503f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:07.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.625+0000 7ff4a37fe700 1 -- 172.21.15.78:0/3157204282 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff4a406ab40 msgr2=0x7ff4a41ea440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:07.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.625+0000 7ff4a37fe700 1 --2- 172.21.15.78:0/3157204282 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff4a406ab40 0x7ff4a41ea440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:07.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.625+0000 7ff4a37fe700 1 -- 172.21.15.78:0/3157204282 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff4a40c0350 msgr2=0x7ff4a41503f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:07.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.625+0000 7ff4a37fe700 1 --2- 172.21.15.78:0/3157204282 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff4a40c0350 0x7ff4a41503f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:07.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.625+0000 7ff4a37fe700 1 -- 172.21.15.78:0/3157204282 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff49c009a70 con 0x7ff4a406f1e0 2023-10-14T11:59:07.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.625+0000 7ff4a3fff700 1 --2- 172.21.15.78:0/3157204282 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff4a406ab40 0x7ff4a41ea440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:59:07.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.625+0000 7ff4a37fe700 1 --2- 172.21.15.78:0/3157204282 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4a406f1e0 0x7ff4a4153a20 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7ff49800c7f0 tx=0x7ff49800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:07.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.625+0000 7ff4a0ff9700 1 -- 172.21.15.78:0/3157204282 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff498011800 con 0x7ff4a406f1e0 2023-10-14T11:59:07.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.625+0000 7ff4a0ff9700 1 -- 172.21.15.78:0/3157204282 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff498011960 con 0x7ff4a406f1e0 2023-10-14T11:59:07.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.625+0000 7ff4aa538700 1 -- 172.21.15.78:0/3157204282 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4a41ea980 con 0x7ff4a406f1e0 2023-10-14T11:59:07.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.625+0000 7ff4a0ff9700 1 -- 172.21.15.78:0/3157204282 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff498011cf0 con 0x7ff4a406f1e0 2023-10-14T11:59:07.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.625+0000 7ff4aa538700 1 -- 172.21.15.78:0/3157204282 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4a40c6900 con 0x7ff4a406f1e0 2023-10-14T11:59:07.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.626+0000 7ff4aa538700 1 -- 172.21.15.78:0/3157204282 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4a40c51e0 con 0x7ff4a406f1e0 2023-10-14T11:59:07.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.627+0000 7ff4a0ff9700 1 -- 172.21.15.78:0/3157204282 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff498018360 con 0x7ff4a406f1e0 2023-10-14T11:59:07.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.627+0000 7ff4a0ff9700 1 --2- 172.21.15.78:0/3157204282 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff49006af00 0x7ff49006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:07.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.628+0000 7ff4a3fff700 1 --2- 172.21.15.78:0/3157204282 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff49006af00 0x7ff49006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:07.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.628+0000 7ff4a0ff9700 1 -- 172.21.15.78:0/3157204282 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7ff498089470 con 0x7ff4a406f1e0 2023-10-14T11:59:07.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.628+0000 7ff4a3fff700 1 --2- 172.21.15.78:0/3157204282 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff49006af00 0x7ff49006d3c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff494005910 tx=0x7ff494009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:07.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.632+0000 7ff4a0ff9700 1 -- 172.21.15.78:0/3157204282 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff4a40c51e0 con 0x7ff4a406f1e0 2023-10-14T11:59:07.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.816+0000 7ff4aa538700 1 -- 172.21.15.78:0/3157204282 --> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff4a4154a00 con 0x7ff49006af00 2023-10-14T11:59:07.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.823+0000 7ff4a0ff9700 1 -- 172.21.15.78:0/3157204282 <== mgr.4104 v2:172.21.15.78:6832/3171890074 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+263555 (secure 0 0 0) 0x7ff4940014f0 con 0x7ff49006af00 2023-10-14T11:59:07.830 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:59:07.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.826+0000 7ff4aa538700 1 -- 172.21.15.78:0/3157204282 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff49006af00 msgr2=0x7ff49006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:07.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.826+0000 7ff4aa538700 1 --2- 172.21.15.78:0/3157204282 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff49006af00 0x7ff49006d3c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff494005910 tx=0x7ff494009040 comp rx=0 tx=0).stop 2023-10-14T11:59:07.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.826+0000 7ff4aa538700 1 -- 172.21.15.78:0/3157204282 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4a406f1e0 msgr2=0x7ff4a4153a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:07.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.826+0000 7ff4aa538700 1 --2- 172.21.15.78:0/3157204282 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4a406f1e0 0x7ff4a4153a20 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7ff49800c7f0 tx=0x7ff49800cbb0 comp rx=0 tx=0).stop 2023-10-14T11:59:07.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.826+0000 7ff4aa538700 1 -- 172.21.15.78:0/3157204282 shutdown_connections 2023-10-14T11:59:07.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.826+0000 7ff4aa538700 1 --2- 172.21.15.78:0/3157204282 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff49006af00 0x7ff49006d3c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:07.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.826+0000 7ff4aa538700 1 --2- 172.21.15.78:0/3157204282 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff4a40c0350 0x7ff4a41503f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:07.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.826+0000 7ff4aa538700 1 --2- 172.21.15.78:0/3157204282 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff4a406ab40 0x7ff4a41ea440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:07.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.826+0000 7ff4aa538700 1 --2- 172.21.15.78:0/3157204282 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4a406f1e0 0x7ff4a4153a20 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:07.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.826+0000 7ff4aa538700 1 -- 172.21.15.78:0/3157204282 >> 172.21.15.78:0/3157204282 conn(0x7ff4a40c3ac0 msgr2=0x7ff4a40c21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:07.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.826+0000 7ff4aa538700 1 -- 172.21.15.78:0/3157204282 shutdown_connections 2023-10-14T11:59:07.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:07.826+0000 7ff4aa538700 1 -- 172.21.15.78:0/3157204282 wait complete. 2023-10-14T11:59:07.835 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2023-10-14T11:59:07.840 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":55,"stamp":"2023-10-14T11:59:06.975942+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":73,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218932,"kb_used_data":2764,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755788,"statfs":{"total":773094113280,"available":772869926912,"internally_reserved":0,"allocated":2830336,"data_stored":2193229,"data_compressed":65244,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001480"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.143338+0000","last_change":"2023-10-14T11:59:02.143338+0000","last_active":"2023-10-14T11:59:02.143338+0000","last_peered":"2023-10-14T11:59:02.143338+0000","last_clean":"2023-10-14T11:59:02.143338+0000","last_became_active":"2023-10-14T11:59:02.143049+0000","last_became_peered":"2023-10-14T11:59:02.143049+0000","last_unstale":"2023-10-14T11:59:02.143338+0000","last_undegraded":"2023-10-14T11:59:02.143338+0000","last_fullsized":"2023-10-14T11:59:02.143338+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:43:53.205090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":4,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.139597+0000","last_change":"2023-10-14T11:59:02.138579+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.139597+0000","last_undegraded":"2023-10-14T11:59:02.139597+0000","last_fullsized":"2023-10-14T11:59:02.139597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:35:59.785182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.150827+0000","last_change":"2023-10-14T11:59:02.144594+0000","last_active":"2023-10-14T11:59:02.150827+0000","last_peered":"2023-10-14T11:59:02.150827+0000","last_clean":"2023-10-14T11:59:02.150827+0000","last_became_active":"2023-10-14T11:59:02.144322+0000","last_became_peered":"2023-10-14T11:59:02.144322+0000","last_unstale":"2023-10-14T11:59:02.150827+0000","last_undegraded":"2023-10-14T11:59:02.150827+0000","last_fullsized":"2023-10-14T11:59:02.150827+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:08:25.866669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.146281+0000","last_change":"2023-10-14T11:59:02.146281+0000","last_active":"2023-10-14T11:59:02.146281+0000","last_peered":"2023-10-14T11:59:02.146281+0000","last_clean":"2023-10-14T11:59:02.146281+0000","last_became_active":"2023-10-14T11:59:02.146015+0000","last_became_peered":"2023-10-14T11:59:02.146015+0000","last_unstale":"2023-10-14T11:59:02.146281+0000","last_undegraded":"2023-10-14T11:59:02.146281+0000","last_fullsized":"2023-10-14T11:59:02.146281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:54:28.849250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.140518+0000","last_change":"2023-10-14T11:59:02.140518+0000","last_active":"2023-10-14T11:59:02.140518+0000","last_peered":"2023-10-14T11:59:02.140518+0000","last_clean":"2023-10-14T11:59:02.140518+0000","last_became_active":"2023-10-14T11:59:02.140238+0000","last_became_peered":"2023-10-14T11:59:02.140238+0000","last_unstale":"2023-10-14T11:59:02.140518+0000","last_undegraded":"2023-10-14T11:59:02.140518+0000","last_fullsized":"2023-10-14T11:59:02.140518+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:43:35.020618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.136431+0000","last_change":"2023-10-14T11:59:02.136431+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.136431+0000","last_undegraded":"2023-10-14T11:59:02.136431+0000","last_fullsized":"2023-10-14T11:59:02.136431+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:08:25.866669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.135963+0000","last_change":"2023-10-14T11:59:02.135963+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.135963+0000","last_undegraded":"2023-10-14T11:59:02.135963+0000","last_fullsized":"2023-10-14T11:59:02.135963+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:05:38.808781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.135986+0000","last_change":"2023-10-14T11:59:02.135986+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.135986+0000","last_undegraded":"2023-10-14T11:59:02.135986+0000","last_fullsized":"2023-10-14T11:59:02.135986+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:57:18.507681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.132676+0000","last_change":"2023-10-14T11:59:02.132676+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.132676+0000","last_undegraded":"2023-10-14T11:59:02.132676+0000","last_fullsized":"2023-10-14T11:59:02.132676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:20:02.525878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.133724+0000","last_change":"2023-10-14T11:59:02.133724+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.133724+0000","last_undegraded":"2023-10-14T11:59:02.133724+0000","last_fullsized":"2023-10-14T11:59:02.133724+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:15:00.593091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.143474+0000","last_change":"2023-10-14T11:59:02.143474+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.143474+0000","last_undegraded":"2023-10-14T11:59:02.143474+0000","last_fullsized":"2023-10-14T11:59:02.143474+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:07:15.007617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.143255+0000","last_change":"2023-10-14T11:59:02.139674+0000","last_active":"2023-10-14T11:59:02.143255+0000","last_peered":"2023-10-14T11:59:02.143255+0000","last_clean":"2023-10-14T11:59:02.143255+0000","last_became_active":"2023-10-14T11:59:02.139404+0000","last_became_peered":"2023-10-14T11:59:02.139404+0000","last_unstale":"2023-10-14T11:59:02.143255+0000","last_undegraded":"2023-10-14T11:59:02.143255+0000","last_fullsized":"2023-10-14T11:59:02.143255+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:30:41.965448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.145057+0000","last_change":"2023-10-14T11:59:02.145057+0000","last_active":"2023-10-14T11:59:02.145057+0000","last_peered":"2023-10-14T11:59:02.145057+0000","last_clean":"2023-10-14T11:59:02.145057+0000","last_became_active":"2023-10-14T11:59:02.144776+0000","last_became_peered":"2023-10-14T11:59:02.144776+0000","last_unstale":"2023-10-14T11:59:02.145057+0000","last_undegraded":"2023-10-14T11:59:02.145057+0000","last_fullsized":"2023-10-14T11:59:02.145057+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:03:39.032281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.137158+0000","last_change":"2023-10-14T11:59:02.137158+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.137158+0000","last_undegraded":"2023-10-14T11:59:02.137158+0000","last_fullsized":"2023-10-14T11:59:02.137158+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:12:09.205507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.140788+0000","last_change":"2023-10-14T11:59:02.134707+0000","last_active":"2023-10-14T11:59:02.140788+0000","last_peered":"2023-10-14T11:59:02.140788+0000","last_clean":"2023-10-14T11:59:02.140788+0000","last_became_active":"2023-10-14T11:59:02.134424+0000","last_became_peered":"2023-10-14T11:59:02.134424+0000","last_unstale":"2023-10-14T11:59:02.140788+0000","last_undegraded":"2023-10-14T11:59:02.140788+0000","last_fullsized":"2023-10-14T11:59:02.140788+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:14.681110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.138148+0000","last_change":"2023-10-14T11:59:02.138148+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.138148+0000","last_undegraded":"2023-10-14T11:59:02.138148+0000","last_fullsized":"2023-10-14T11:59:02.138148+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:27:07.788319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.136715+0000","last_change":"2023-10-14T11:59:02.136715+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.136715+0000","last_undegraded":"2023-10-14T11:59:02.136715+0000","last_fullsized":"2023-10-14T11:59:02.136715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:30:41.965448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.135337+0000","last_change":"2023-10-14T11:59:02.135337+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.135337+0000","last_undegraded":"2023-10-14T11:59:02.135337+0000","last_fullsized":"2023-10-14T11:59:02.135337+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:05:38.808781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.141545+0000","last_change":"2023-10-14T11:59:02.141545+0000","last_active":"2023-10-14T11:59:02.141545+0000","last_peered":"2023-10-14T11:59:02.141545+0000","last_clean":"2023-10-14T11:59:02.141545+0000","last_became_active":"2023-10-14T11:59:02.141281+0000","last_became_peered":"2023-10-14T11:59:02.141281+0000","last_unstale":"2023-10-14T11:59:02.141545+0000","last_undegraded":"2023-10-14T11:59:02.141545+0000","last_fullsized":"2023-10-14T11:59:02.141545+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:26.625395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.145736+0000","last_change":"2023-10-14T11:59:02.145736+0000","last_active":"2023-10-14T11:59:02.145736+0000","last_peered":"2023-10-14T11:59:02.145736+0000","last_clean":"2023-10-14T11:59:02.145736+0000","last_became_active":"2023-10-14T11:59:02.145465+0000","last_became_peered":"2023-10-14T11:59:02.145465+0000","last_unstale":"2023-10-14T11:59:02.145736+0000","last_undegraded":"2023-10-14T11:59:02.145736+0000","last_fullsized":"2023-10-14T11:59:02.145736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:08:45.915417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.132115+0000","last_change":"2023-10-14T11:59:02.132115+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.132115+0000","last_undegraded":"2023-10-14T11:59:02.132115+0000","last_fullsized":"2023-10-14T11:59:02.132115+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:53:44.574661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.132815+0000","last_change":"2023-10-14T11:59:02.132815+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.132815+0000","last_undegraded":"2023-10-14T11:59:02.132815+0000","last_fullsized":"2023-10-14T11:59:02.132815+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:39:49.443148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.137732+0000","last_change":"2023-10-14T11:59:02.137732+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.137732+0000","last_undegraded":"2023-10-14T11:59:02.137732+0000","last_fullsized":"2023-10-14T11:59:02.137732+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:54:28.849250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.150417+0000","last_change":"2023-10-14T11:59:02.149221+0000","last_active":"2023-10-14T11:59:02.150417+0000","last_peered":"2023-10-14T11:59:02.150417+0000","last_clean":"2023-10-14T11:59:02.150417+0000","last_became_active":"2023-10-14T11:59:02.148952+0000","last_became_peered":"2023-10-14T11:59:02.148952+0000","last_unstale":"2023-10-14T11:59:02.150417+0000","last_undegraded":"2023-10-14T11:59:02.150417+0000","last_fullsized":"2023-10-14T11:59:02.150417+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:58:07.847931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.146060+0000","last_change":"2023-10-14T11:59:02.146060+0000","last_active":"2023-10-14T11:59:02.146060+0000","last_peered":"2023-10-14T11:59:02.146060+0000","last_clean":"2023-10-14T11:59:02.146060+0000","last_became_active":"2023-10-14T11:59:02.145805+0000","last_became_peered":"2023-10-14T11:59:02.145805+0000","last_unstale":"2023-10-14T11:59:02.146060+0000","last_undegraded":"2023-10-14T11:59:02.146060+0000","last_fullsized":"2023-10-14T11:59:02.146060+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:39:49.443148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.132003+0000","last_change":"2023-10-14T11:59:02.132003+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.132003+0000","last_undegraded":"2023-10-14T11:59:02.132003+0000","last_fullsized":"2023-10-14T11:59:02.132003+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:54:28.849250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.146857+0000","last_change":"2023-10-14T11:59:02.146857+0000","last_active":"2023-10-14T11:59:02.146857+0000","last_peered":"2023-10-14T11:59:02.146857+0000","last_clean":"2023-10-14T11:59:02.146857+0000","last_became_active":"2023-10-14T11:59:02.146627+0000","last_became_peered":"2023-10-14T11:59:02.146627+0000","last_unstale":"2023-10-14T11:59:02.146857+0000","last_undegraded":"2023-10-14T11:59:02.146857+0000","last_fullsized":"2023-10-14T11:59:02.146857+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:30:41.965448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.149774+0000","last_change":"2023-10-14T11:59:02.143747+0000","last_active":"2023-10-14T11:59:02.149774+0000","last_peered":"2023-10-14T11:59:02.149774+0000","last_clean":"2023-10-14T11:59:02.149774+0000","last_became_active":"2023-10-14T11:59:02.143421+0000","last_became_peered":"2023-10-14T11:59:02.143421+0000","last_unstale":"2023-10-14T11:59:02.149774+0000","last_undegraded":"2023-10-14T11:59:02.149774+0000","last_fullsized":"2023-10-14T11:59:02.149774+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:27:07.788319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.139034+0000","last_change":"2023-10-14T11:59:02.139034+0000","last_active":"2023-10-14T11:59:02.139034+0000","last_peered":"2023-10-14T11:59:02.139034+0000","last_clean":"2023-10-14T11:59:02.139034+0000","last_became_active":"2023-10-14T11:59:02.138730+0000","last_became_peered":"2023-10-14T11:59:02.138730+0000","last_unstale":"2023-10-14T11:59:02.139034+0000","last_undegraded":"2023-10-14T11:59:02.139034+0000","last_fullsized":"2023-10-14T11:59:02.139034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:48:37.789205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.136790+0000","last_change":"2023-10-14T11:59:02.136790+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.136790+0000","last_undegraded":"2023-10-14T11:59:02.136790+0000","last_fullsized":"2023-10-14T11:59:02.136790+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:58:07.847931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.131455+0000","last_change":"2023-10-14T11:59:02.131455+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.131455+0000","last_undegraded":"2023-10-14T11:59:02.131455+0000","last_fullsized":"2023-10-14T11:59:02.131455+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:05:58.896548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.138555+0000","last_change":"2023-10-14T11:59:02.138555+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.138555+0000","last_undegraded":"2023-10-14T11:59:02.138555+0000","last_fullsized":"2023-10-14T11:59:02.138555+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:08:25.866669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.136873+0000","last_change":"2023-10-14T11:59:02.136873+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.136873+0000","last_undegraded":"2023-10-14T11:59:02.136873+0000","last_fullsized":"2023-10-14T11:59:02.136873+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:20:02.525878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.133827+0000","last_change":"2023-10-14T11:59:01.170225+0000","last_active":"2023-10-14T11:59:02.133827+0000","last_peered":"2023-10-14T11:59:02.133827+0000","last_clean":"2023-10-14T11:59:02.133827+0000","last_became_active":"2023-10-14T11:59:01.169927+0000","last_became_peered":"2023-10-14T11:59:01.169927+0000","last_unstale":"2023-10-14T11:59:02.133827+0000","last_undegraded":"2023-10-14T11:59:02.133827+0000","last_fullsized":"2023-10-14T11:59:02.133827+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:25.623744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.139277+0000","last_change":"2023-10-14T11:59:02.139277+0000","last_active":"2023-10-14T11:59:02.139277+0000","last_peered":"2023-10-14T11:59:02.139277+0000","last_clean":"2023-10-14T11:59:02.139277+0000","last_became_active":"2023-10-14T11:59:02.138985+0000","last_became_peered":"2023-10-14T11:59:02.138985+0000","last_unstale":"2023-10-14T11:59:02.139277+0000","last_undegraded":"2023-10-14T11:59:02.139277+0000","last_fullsized":"2023-10-14T11:59:02.139277+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:15:00.593091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.137138+0000","last_change":"2023-10-14T11:59:01.178406+0000","last_active":"2023-10-14T11:59:02.137138+0000","last_peered":"2023-10-14T11:59:02.137138+0000","last_clean":"2023-10-14T11:59:02.137138+0000","last_became_active":"2023-10-14T11:59:01.178136+0000","last_became_peered":"2023-10-14T11:59:01.178136+0000","last_unstale":"2023-10-14T11:59:02.137138+0000","last_undegraded":"2023-10-14T11:59:02.137138+0000","last_fullsized":"2023-10-14T11:59:02.137138+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:43:52.203439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.139738+0000","last_change":"2023-10-14T11:59:02.139738+0000","last_active":"2023-10-14T11:59:02.139738+0000","last_peered":"2023-10-14T11:59:02.139738+0000","last_clean":"2023-10-14T11:59:02.139738+0000","last_became_active":"2023-10-14T11:59:02.139431+0000","last_became_peered":"2023-10-14T11:59:02.139431+0000","last_unstale":"2023-10-14T11:59:02.139738+0000","last_undegraded":"2023-10-14T11:59:02.139738+0000","last_fullsized":"2023-10-14T11:59:02.139738+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:41:47.404184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.133347+0000","last_change":"2023-10-14T11:59:01.179865+0000","last_active":"2023-10-14T11:59:02.133347+0000","last_peered":"2023-10-14T11:59:02.133347+0000","last_clean":"2023-10-14T11:59:02.133347+0000","last_became_active":"2023-10-14T11:59:01.178404+0000","last_became_peered":"2023-10-14T11:59:01.178404+0000","last_unstale":"2023-10-14T11:59:02.133347+0000","last_undegraded":"2023-10-14T11:59:02.133347+0000","last_fullsized":"2023-10-14T11:59:02.133347+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:30:40.963797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.146195+0000","last_change":"2023-10-14T11:59:02.146195+0000","last_active":"2023-10-14T11:59:02.146195+0000","last_peered":"2023-10-14T11:59:02.146195+0000","last_clean":"2023-10-14T11:59:02.146195+0000","last_became_active":"2023-10-14T11:59:02.144184+0000","last_became_peered":"2023-10-14T11:59:02.144184+0000","last_unstale":"2023-10-14T11:59:02.146195+0000","last_undegraded":"2023-10-14T11:59:02.146195+0000","last_fullsized":"2023-10-14T11:59:02.146195+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:43:53.205090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.135778+0000","last_change":"2023-10-14T11:59:01.185486+0000","last_active":"2023-10-14T11:59:02.135778+0000","last_peered":"2023-10-14T11:59:02.135778+0000","last_clean":"2023-10-14T11:59:02.135778+0000","last_became_active":"2023-10-14T11:59:01.185200+0000","last_became_peered":"2023-10-14T11:59:01.185200+0000","last_unstale":"2023-10-14T11:59:02.135778+0000","last_undegraded":"2023-10-14T11:59:02.135778+0000","last_fullsized":"2023-10-14T11:59:02.135778+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:48:36.787554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.139052+0000","last_change":"2023-10-14T11:59:02.139052+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.139052+0000","last_undegraded":"2023-10-14T11:59:02.139052+0000","last_fullsized":"2023-10-14T11:59:02.139052+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:36:52.548523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.132966+0000","last_change":"2023-10-14T11:59:01.178774+0000","last_active":"2023-10-14T11:59:02.132966+0000","last_peered":"2023-10-14T11:59:02.132966+0000","last_clean":"2023-10-14T11:59:02.132966+0000","last_became_active":"2023-10-14T11:59:01.178378+0000","last_became_peered":"2023-10-14T11:59:01.178378+0000","last_unstale":"2023-10-14T11:59:02.132966+0000","last_undegraded":"2023-10-14T11:59:02.132966+0000","last_fullsized":"2023-10-14T11:59:02.132966+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:25.623744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.149841+0000","last_change":"2023-10-14T11:59:02.143614+0000","last_active":"2023-10-14T11:59:02.149841+0000","last_peered":"2023-10-14T11:59:02.149841+0000","last_clean":"2023-10-14T11:59:02.149841+0000","last_became_active":"2023-10-14T11:59:02.143332+0000","last_became_peered":"2023-10-14T11:59:02.143332+0000","last_unstale":"2023-10-14T11:59:02.149841+0000","last_undegraded":"2023-10-14T11:59:02.149841+0000","last_fullsized":"2023-10-14T11:59:02.149841+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:12:09.205507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.133277+0000","last_change":"2023-10-14T11:59:01.178778+0000","last_active":"2023-10-14T11:59:02.133277+0000","last_peered":"2023-10-14T11:59:02.133277+0000","last_clean":"2023-10-14T11:59:02.133277+0000","last_became_active":"2023-10-14T11:59:01.178440+0000","last_became_peered":"2023-10-14T11:59:01.178440+0000","last_unstale":"2023-10-14T11:59:02.133277+0000","last_undegraded":"2023-10-14T11:59:02.133277+0000","last_fullsized":"2023-10-14T11:59:02.133277+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:18:16.224262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.139185+0000","last_change":"2023-10-14T11:59:02.139185+0000","last_active":"2023-10-14T11:59:02.139185+0000","last_peered":"2023-10-14T11:59:02.139185+0000","last_clean":"2023-10-14T11:59:02.139185+0000","last_became_active":"2023-10-14T11:59:02.138862+0000","last_became_peered":"2023-10-14T11:59:02.138862+0000","last_unstale":"2023-10-14T11:59:02.139185+0000","last_undegraded":"2023-10-14T11:59:02.139185+0000","last_fullsized":"2023-10-14T11:59:02.139185+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:48:45.563805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.132488+0000","last_change":"2023-10-14T11:59:01.175662+0000","last_active":"2023-10-14T11:59:02.132488+0000","last_peered":"2023-10-14T11:59:02.132488+0000","last_clean":"2023-10-14T11:59:02.132488+0000","last_became_active":"2023-10-14T11:59:01.175405+0000","last_became_peered":"2023-10-14T11:59:01.175405+0000","last_unstale":"2023-10-14T11:59:02.132488+0000","last_undegraded":"2023-10-14T11:59:02.132488+0000","last_fullsized":"2023-10-14T11:59:02.132488+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:57:17.506030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.143559+0000","last_change":"2023-10-14T11:59:02.143559+0000","last_active":"2023-10-14T11:59:02.143559+0000","last_peered":"2023-10-14T11:59:02.143559+0000","last_clean":"2023-10-14T11:59:02.143559+0000","last_became_active":"2023-10-14T11:59:02.143303+0000","last_became_peered":"2023-10-14T11:59:02.143303+0000","last_unstale":"2023-10-14T11:59:02.143559+0000","last_undegraded":"2023-10-14T11:59:02.143559+0000","last_fullsized":"2023-10-14T11:59:02.143559+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:20:02.525878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.132854+0000","last_change":"2023-10-14T11:59:01.167233+0000","last_active":"2023-10-14T11:59:02.132854+0000","last_peered":"2023-10-14T11:59:02.132854+0000","last_clean":"2023-10-14T11:59:02.132854+0000","last_became_active":"2023-10-14T11:59:01.166905+0000","last_became_peered":"2023-10-14T11:59:01.166905+0000","last_unstale":"2023-10-14T11:59:02.132854+0000","last_undegraded":"2023-10-14T11:59:02.132854+0000","last_fullsized":"2023-10-14T11:59:02.132854+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:43:52.203439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.152247+0000","last_change":"2023-10-14T11:59:02.150665+0000","last_active":"2023-10-14T11:59:02.152247+0000","last_peered":"2023-10-14T11:59:02.152247+0000","last_clean":"2023-10-14T11:59:02.152247+0000","last_became_active":"2023-10-14T11:59:02.150384+0000","last_became_peered":"2023-10-14T11:59:02.150384+0000","last_unstale":"2023-10-14T11:59:02.152247+0000","last_undegraded":"2023-10-14T11:59:02.152247+0000","last_fullsized":"2023-10-14T11:59:02.152247+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:07:15.007617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.133851+0000","last_change":"2023-10-14T11:59:01.179314+0000","last_active":"2023-10-14T11:59:02.133851+0000","last_peered":"2023-10-14T11:59:02.133851+0000","last_clean":"2023-10-14T11:59:02.133851+0000","last_became_active":"2023-10-14T11:59:01.179019+0000","last_became_peered":"2023-10-14T11:59:01.179019+0000","last_unstale":"2023-10-14T11:59:02.133851+0000","last_undegraded":"2023-10-14T11:59:02.133851+0000","last_fullsized":"2023-10-14T11:59:02.133851+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:27:06.786668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.137973+0000","last_change":"2023-10-14T11:59:02.137973+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.137973+0000","last_undegraded":"2023-10-14T11:59:02.137973+0000","last_fullsized":"2023-10-14T11:59:02.137973+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:36:52.548523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.137264+0000","last_change":"2023-10-14T11:59:01.177873+0000","last_active":"2023-10-14T11:59:02.137264+0000","last_peered":"2023-10-14T11:59:02.137264+0000","last_clean":"2023-10-14T11:59:02.137264+0000","last_became_active":"2023-10-14T11:59:01.177594+0000","last_became_peered":"2023-10-14T11:59:01.177594+0000","last_unstale":"2023-10-14T11:59:02.137264+0000","last_undegraded":"2023-10-14T11:59:02.137264+0000","last_fullsized":"2023-10-14T11:59:02.137264+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:25.623744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.132187+0000","last_change":"2023-10-14T11:59:02.132187+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.132187+0000","last_undegraded":"2023-10-14T11:59:02.132187+0000","last_fullsized":"2023-10-14T11:59:02.132187+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:47:13.361893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.135909+0000","last_change":"2023-10-14T11:59:01.178143+0000","last_active":"2023-10-14T11:59:02.135909+0000","last_peered":"2023-10-14T11:59:02.135909+0000","last_clean":"2023-10-14T11:59:02.135909+0000","last_became_active":"2023-10-14T11:59:01.177736+0000","last_became_peered":"2023-10-14T11:59:01.177736+0000","last_unstale":"2023-10-14T11:59:02.135909+0000","last_undegraded":"2023-10-14T11:59:02.135909+0000","last_fullsized":"2023-10-14T11:59:02.135909+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:35:58.783531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.150847+0000","last_change":"2023-10-14T11:59:02.145816+0000","last_active":"2023-10-14T11:59:02.150847+0000","last_peered":"2023-10-14T11:59:02.150847+0000","last_clean":"2023-10-14T11:59:02.150847+0000","last_became_active":"2023-10-14T11:59:02.145548+0000","last_became_peered":"2023-10-14T11:59:02.145548+0000","last_unstale":"2023-10-14T11:59:02.150847+0000","last_undegraded":"2023-10-14T11:59:02.150847+0000","last_fullsized":"2023-10-14T11:59:02.150847+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:13:03.743718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.133733+0000","last_change":"2023-10-14T11:59:01.180187+0000","last_active":"2023-10-14T11:59:02.133733+0000","last_peered":"2023-10-14T11:59:02.133733+0000","last_clean":"2023-10-14T11:59:02.133733+0000","last_became_active":"2023-10-14T11:59:01.179909+0000","last_became_peered":"2023-10-14T11:59:01.179909+0000","last_unstale":"2023-10-14T11:59:02.133733+0000","last_undegraded":"2023-10-14T11:59:02.133733+0000","last_fullsized":"2023-10-14T11:59:02.133733+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:58:06.846280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.143696+0000","last_change":"2023-10-14T11:59:02.143696+0000","last_active":"2023-10-14T11:59:02.143696+0000","last_peered":"2023-10-14T11:59:02.143696+0000","last_clean":"2023-10-14T11:59:02.143696+0000","last_became_active":"2023-10-14T11:59:02.143443+0000","last_became_peered":"2023-10-14T11:59:02.143443+0000","last_unstale":"2023-10-14T11:59:02.143696+0000","last_undegraded":"2023-10-14T11:59:02.143696+0000","last_fullsized":"2023-10-14T11:59:02.143696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:47:13.361893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.134332+0000","last_change":"2023-10-14T11:59:01.165901+0000","last_active":"2023-10-14T11:59:02.134332+0000","last_peered":"2023-10-14T11:59:02.134332+0000","last_clean":"2023-10-14T11:59:02.134332+0000","last_became_active":"2023-10-14T11:59:01.165597+0000","last_became_peered":"2023-10-14T11:59:01.165597+0000","last_unstale":"2023-10-14T11:59:02.134332+0000","last_undegraded":"2023-10-14T11:59:02.134332+0000","last_fullsized":"2023-10-14T11:59:02.134332+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:05:37.807130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.141427+0000","last_change":"2023-10-14T11:59:02.141427+0000","last_active":"2023-10-14T11:59:02.141427+0000","last_peered":"2023-10-14T11:59:02.141427+0000","last_clean":"2023-10-14T11:59:02.141427+0000","last_became_active":"2023-10-14T11:59:02.141166+0000","last_became_peered":"2023-10-14T11:59:02.141166+0000","last_unstale":"2023-10-14T11:59:02.141427+0000","last_undegraded":"2023-10-14T11:59:02.141427+0000","last_fullsized":"2023-10-14T11:59:02.141427+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:18:17.225913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.136387+0000","last_change":"2023-10-14T11:59:01.172037+0000","last_active":"2023-10-14T11:59:02.136387+0000","last_peered":"2023-10-14T11:59:02.136387+0000","last_clean":"2023-10-14T11:59:02.136387+0000","last_became_active":"2023-10-14T11:59:01.170760+0000","last_became_peered":"2023-10-14T11:59:01.170760+0000","last_unstale":"2023-10-14T11:59:02.136387+0000","last_undegraded":"2023-10-14T11:59:02.136387+0000","last_fullsized":"2023-10-14T11:59:02.136387+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:57:17.506030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.138925+0000","last_change":"2023-10-14T11:59:02.138925+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.138925+0000","last_undegraded":"2023-10-14T11:59:02.138925+0000","last_fullsized":"2023-10-14T11:59:02.138925+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:12:09.205507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.132642+0000","last_change":"2023-10-14T11:59:01.178960+0000","last_active":"2023-10-14T11:59:02.132642+0000","last_peered":"2023-10-14T11:59:02.132642+0000","last_clean":"2023-10-14T11:59:02.132642+0000","last_became_active":"2023-10-14T11:59:01.178691+0000","last_became_peered":"2023-10-14T11:59:01.178691+0000","last_unstale":"2023-10-14T11:59:02.132642+0000","last_undegraded":"2023-10-14T11:59:02.132642+0000","last_fullsized":"2023-10-14T11:59:02.132642+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:13.679459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.140181+0000","last_change":"2023-10-14T11:59:02.140181+0000","last_active":"2023-10-14T11:59:02.140181+0000","last_peered":"2023-10-14T11:59:02.140181+0000","last_clean":"2023-10-14T11:59:02.140181+0000","last_became_active":"2023-10-14T11:59:02.139382+0000","last_became_peered":"2023-10-14T11:59:02.139382+0000","last_unstale":"2023-10-14T11:59:02.140181+0000","last_undegraded":"2023-10-14T11:59:02.140181+0000","last_fullsized":"2023-10-14T11:59:02.140181+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:20:02.525878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.138336+0000","last_change":"2023-10-14T11:59:01.178769+0000","last_active":"2023-10-14T11:59:02.138336+0000","last_peered":"2023-10-14T11:59:02.138336+0000","last_clean":"2023-10-14T11:59:02.138336+0000","last_became_active":"2023-10-14T11:59:01.178288+0000","last_became_peered":"2023-10-14T11:59:01.178288+0000","last_unstale":"2023-10-14T11:59:02.138336+0000","last_undegraded":"2023-10-14T11:59:02.138336+0000","last_fullsized":"2023-10-14T11:59:02.138336+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:07:14.005966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.132368+0000","last_change":"2023-10-14T11:59:02.132368+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.132368+0000","last_undegraded":"2023-10-14T11:59:02.132368+0000","last_fullsized":"2023-10-14T11:59:02.132368+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:05:58.896548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.134614+0000","last_change":"2023-10-14T11:59:01.183630+0000","last_active":"2023-10-14T11:59:02.134614+0000","last_peered":"2023-10-14T11:59:02.134614+0000","last_clean":"2023-10-14T11:59:02.134614+0000","last_became_active":"2023-10-14T11:59:01.183408+0000","last_became_peered":"2023-10-14T11:59:01.183408+0000","last_unstale":"2023-10-14T11:59:02.134614+0000","last_undegraded":"2023-10-14T11:59:02.134614+0000","last_fullsized":"2023-10-14T11:59:02.134614+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:25.623744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.145545+0000","last_change":"2023-10-14T11:59:02.141878+0000","last_active":"2023-10-14T11:59:02.145545+0000","last_peered":"2023-10-14T11:59:02.145545+0000","last_clean":"2023-10-14T11:59:02.145545+0000","last_became_active":"2023-10-14T11:59:02.141229+0000","last_became_peered":"2023-10-14T11:59:02.141229+0000","last_unstale":"2023-10-14T11:59:02.145545+0000","last_undegraded":"2023-10-14T11:59:02.145545+0000","last_fullsized":"2023-10-14T11:59:02.145545+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:11:18.803472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.132715+0000","last_change":"2023-10-14T11:59:01.165942+0000","last_active":"2023-10-14T11:59:02.132715+0000","last_peered":"2023-10-14T11:59:02.132715+0000","last_clean":"2023-10-14T11:59:02.132715+0000","last_became_active":"2023-10-14T11:59:01.165646+0000","last_became_peered":"2023-10-14T11:59:01.165646+0000","last_unstale":"2023-10-14T11:59:02.132715+0000","last_undegraded":"2023-10-14T11:59:02.132715+0000","last_fullsized":"2023-10-14T11:59:02.132715+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:14:59.591440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.147269+0000","last_change":"2023-10-14T11:59:02.147269+0000","last_active":"2023-10-14T11:59:02.147269+0000","last_peered":"2023-10-14T11:59:02.147269+0000","last_clean":"2023-10-14T11:59:02.147269+0000","last_became_active":"2023-10-14T11:59:02.147005+0000","last_became_peered":"2023-10-14T11:59:02.147005+0000","last_unstale":"2023-10-14T11:59:02.147269+0000","last_undegraded":"2023-10-14T11:59:02.147269+0000","last_fullsized":"2023-10-14T11:59:02.147269+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:18:17.225913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.131810+0000","last_change":"2023-10-14T11:59:01.183080+0000","last_active":"2023-10-14T11:59:02.131810+0000","last_peered":"2023-10-14T11:59:02.131810+0000","last_clean":"2023-10-14T11:59:02.131810+0000","last_became_active":"2023-10-14T11:59:01.182786+0000","last_became_peered":"2023-10-14T11:59:01.182786+0000","last_unstale":"2023-10-14T11:59:02.131810+0000","last_undegraded":"2023-10-14T11:59:02.131810+0000","last_fullsized":"2023-10-14T11:59:02.131810+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:05:27.508197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.146618+0000","last_change":"2023-10-14T11:59:02.146618+0000","last_active":"2023-10-14T11:59:02.146618+0000","last_peered":"2023-10-14T11:59:02.146618+0000","last_clean":"2023-10-14T11:59:02.146618+0000","last_became_active":"2023-10-14T11:59:02.146352+0000","last_became_peered":"2023-10-14T11:59:02.146352+0000","last_unstale":"2023-10-14T11:59:02.146618+0000","last_undegraded":"2023-10-14T11:59:02.146618+0000","last_fullsized":"2023-10-14T11:59:02.146618+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:48:45.563805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.132063+0000","last_change":"2023-10-14T11:59:01.178771+0000","last_active":"2023-10-14T11:59:02.132063+0000","last_peered":"2023-10-14T11:59:02.132063+0000","last_clean":"2023-10-14T11:59:02.132063+0000","last_became_active":"2023-10-14T11:59:01.178416+0000","last_became_peered":"2023-10-14T11:59:01.178416+0000","last_unstale":"2023-10-14T11:59:02.132063+0000","last_undegraded":"2023-10-14T11:59:02.132063+0000","last_fullsized":"2023-10-14T11:59:02.132063+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:11:46.659890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.145624+0000","last_change":"2023-10-14T11:59:02.145624+0000","last_active":"2023-10-14T11:59:02.145624+0000","last_peered":"2023-10-14T11:59:02.145624+0000","last_clean":"2023-10-14T11:59:02.145624+0000","last_became_active":"2023-10-14T11:59:02.145381+0000","last_became_peered":"2023-10-14T11:59:02.145381+0000","last_unstale":"2023-10-14T11:59:02.145624+0000","last_undegraded":"2023-10-14T11:59:02.145624+0000","last_fullsized":"2023-10-14T11:59:02.145624+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:21:57.196717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.137214+0000","last_change":"2023-10-14T11:59:01.177556+0000","last_active":"2023-10-14T11:59:02.137214+0000","last_peered":"2023-10-14T11:59:02.137214+0000","last_clean":"2023-10-14T11:59:02.137214+0000","last_became_active":"2023-10-14T11:59:01.177313+0000","last_became_peered":"2023-10-14T11:59:01.177313+0000","last_unstale":"2023-10-14T11:59:02.137214+0000","last_undegraded":"2023-10-14T11:59:02.137214+0000","last_fullsized":"2023-10-14T11:59:02.137214+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:27:02.969220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.141409+0000","last_change":"2023-10-14T11:59:02.141409+0000","last_active":"2023-10-14T11:59:02.141409+0000","last_peered":"2023-10-14T11:59:02.141409+0000","last_clean":"2023-10-14T11:59:02.141409+0000","last_became_active":"2023-10-14T11:59:02.141163+0000","last_became_peered":"2023-10-14T11:59:02.141163+0000","last_unstale":"2023-10-14T11:59:02.141409+0000","last_undegraded":"2023-10-14T11:59:02.141409+0000","last_fullsized":"2023-10-14T11:59:02.141409+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:54:06.957363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.133688+0000","last_change":"2023-10-14T11:59:01.168240+0000","last_active":"2023-10-14T11:59:02.133688+0000","last_peered":"2023-10-14T11:59:02.133688+0000","last_clean":"2023-10-14T11:59:02.133688+0000","last_became_active":"2023-10-14T11:59:01.167944+0000","last_became_peered":"2023-10-14T11:59:01.167944+0000","last_unstale":"2023-10-14T11:59:02.133688+0000","last_undegraded":"2023-10-14T11:59:02.133688+0000","last_fullsized":"2023-10-14T11:59:02.133688+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:27:02.969220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":4,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.140710+0000","last_change":"2023-10-14T11:59:02.133062+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.140710+0000","last_undegraded":"2023-10-14T11:59:02.140710+0000","last_fullsized":"2023-10-14T11:59:02.140710+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:40:14.734059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.133269+0000","last_change":"2023-10-14T11:59:01.176530+0000","last_active":"2023-10-14T11:59:02.133269+0000","last_peered":"2023-10-14T11:59:02.133269+0000","last_clean":"2023-10-14T11:59:02.133269+0000","last_became_active":"2023-10-14T11:59:01.176193+0000","last_became_peered":"2023-10-14T11:59:01.176193+0000","last_unstale":"2023-10-14T11:59:02.133269+0000","last_undegraded":"2023-10-14T11:59:02.133269+0000","last_fullsized":"2023-10-14T11:59:02.133269+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:54:27.847599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.136372+0000","last_change":"2023-10-14T11:59:02.136372+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.136372+0000","last_undegraded":"2023-10-14T11:59:02.136372+0000","last_fullsized":"2023-10-14T11:59:02.136372+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:14.681110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.130535+0000","last_change":"2023-10-14T11:59:01.178157+0000","last_active":"2023-10-14T11:59:02.130535+0000","last_peered":"2023-10-14T11:59:02.130535+0000","last_clean":"2023-10-14T11:59:02.130535+0000","last_became_active":"2023-10-14T11:59:01.177863+0000","last_became_peered":"2023-10-14T11:59:01.177863+0000","last_unstale":"2023-10-14T11:59:02.130535+0000","last_undegraded":"2023-10-14T11:59:02.130535+0000","last_fullsized":"2023-10-14T11:59:02.130535+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:57:17.506030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.143662+0000","last_change":"2023-10-14T11:59:02.143662+0000","last_active":"2023-10-14T11:59:02.143662+0000","last_peered":"2023-10-14T11:59:02.143662+0000","last_clean":"2023-10-14T11:59:02.143662+0000","last_became_active":"2023-10-14T11:59:02.143456+0000","last_became_peered":"2023-10-14T11:59:02.143456+0000","last_unstale":"2023-10-14T11:59:02.143662+0000","last_undegraded":"2023-10-14T11:59:02.143662+0000","last_fullsized":"2023-10-14T11:59:02.143662+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:33:27.410202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.134465+0000","last_change":"2023-10-14T11:59:01.181546+0000","last_active":"2023-10-14T11:59:02.134465+0000","last_peered":"2023-10-14T11:59:02.134465+0000","last_clean":"2023-10-14T11:59:02.134465+0000","last_became_active":"2023-10-14T11:59:01.181255+0000","last_became_peered":"2023-10-14T11:59:01.181255+0000","last_unstale":"2023-10-14T11:59:02.134465+0000","last_undegraded":"2023-10-14T11:59:02.134465+0000","last_fullsized":"2023-10-14T11:59:02.134465+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:27:02.969220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.145565+0000","last_change":"2023-10-14T11:59:02.142119+0000","last_active":"2023-10-14T11:59:02.145565+0000","last_peered":"2023-10-14T11:59:02.145565+0000","last_clean":"2023-10-14T11:59:02.145565+0000","last_became_active":"2023-10-14T11:59:02.141586+0000","last_became_peered":"2023-10-14T11:59:02.141586+0000","last_unstale":"2023-10-14T11:59:02.145565+0000","last_undegraded":"2023-10-14T11:59:02.145565+0000","last_fullsized":"2023-10-14T11:59:02.145565+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:36:52.548523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.137387+0000","last_change":"2023-10-14T11:59:01.180897+0000","last_active":"2023-10-14T11:59:02.137387+0000","last_peered":"2023-10-14T11:59:02.137387+0000","last_clean":"2023-10-14T11:59:02.137387+0000","last_became_active":"2023-10-14T11:59:01.180524+0000","last_became_peered":"2023-10-14T11:59:01.180524+0000","last_unstale":"2023-10-14T11:59:02.137387+0000","last_undegraded":"2023-10-14T11:59:02.137387+0000","last_fullsized":"2023-10-14T11:59:02.137387+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:18:16.224262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.136894+0000","last_change":"2023-10-14T11:59:02.136894+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.136894+0000","last_undegraded":"2023-10-14T11:59:02.136894+0000","last_fullsized":"2023-10-14T11:59:02.136894+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:15:00.593091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.136362+0000","last_change":"2023-10-14T11:59:01.183591+0000","last_active":"2023-10-14T11:59:02.136362+0000","last_peered":"2023-10-14T11:59:02.136362+0000","last_clean":"2023-10-14T11:59:02.136362+0000","last_became_active":"2023-10-14T11:59:01.183019+0000","last_became_peered":"2023-10-14T11:59:01.183019+0000","last_unstale":"2023-10-14T11:59:02.136362+0000","last_undegraded":"2023-10-14T11:59:02.136362+0000","last_fullsized":"2023-10-14T11:59:02.136362+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:05:37.807130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.133356+0000","last_change":"2023-10-14T11:59:02.133356+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.133356+0000","last_undegraded":"2023-10-14T11:59:02.133356+0000","last_fullsized":"2023-10-14T11:59:02.133356+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:39:49.443148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.137080+0000","last_change":"2023-10-14T11:59:01.177539+0000","last_active":"2023-10-14T11:59:02.137080+0000","last_peered":"2023-10-14T11:59:02.137080+0000","last_clean":"2023-10-14T11:59:02.137080+0000","last_became_active":"2023-10-14T11:59:01.177255+0000","last_became_peered":"2023-10-14T11:59:01.177255+0000","last_unstale":"2023-10-14T11:59:02.137080+0000","last_undegraded":"2023-10-14T11:59:02.137080+0000","last_fullsized":"2023-10-14T11:59:02.137080+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:39:48.441497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.137750+0000","last_change":"2023-10-14T11:59:02.137750+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.137750+0000","last_undegraded":"2023-10-14T11:59:02.137750+0000","last_fullsized":"2023-10-14T11:59:02.137750+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:47:13.361893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.136562+0000","last_change":"2023-10-14T11:59:01.182837+0000","last_active":"2023-10-14T11:59:02.136562+0000","last_peered":"2023-10-14T11:59:02.136562+0000","last_clean":"2023-10-14T11:59:02.136562+0000","last_became_active":"2023-10-14T11:59:01.182236+0000","last_became_peered":"2023-10-14T11:59:01.182236+0000","last_unstale":"2023-10-14T11:59:02.136562+0000","last_undegraded":"2023-10-14T11:59:02.136562+0000","last_fullsized":"2023-10-14T11:59:02.136562+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:13.679459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":4,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.138516+0000","last_change":"2023-10-14T11:59:02.138103+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.138516+0000","last_undegraded":"2023-10-14T11:59:02.138516+0000","last_fullsized":"2023-10-14T11:59:02.138516+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:13:03.743718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.133395+0000","last_change":"2023-10-14T11:59:01.168077+0000","last_active":"2023-10-14T11:59:02.133395+0000","last_peered":"2023-10-14T11:59:02.133395+0000","last_clean":"2023-10-14T11:59:02.133395+0000","last_became_active":"2023-10-14T11:59:01.167803+0000","last_became_peered":"2023-10-14T11:59:01.167803+0000","last_unstale":"2023-10-14T11:59:02.133395+0000","last_undegraded":"2023-10-14T11:59:02.133395+0000","last_fullsized":"2023-10-14T11:59:02.133395+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:47:12.360242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":8,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.139387+0000","last_change":"2023-10-14T11:59:02.139387+0000","last_active":"2023-10-14T11:59:02.139387+0000","last_peered":"2023-10-14T11:59:02.139387+0000","last_clean":"2023-10-14T11:59:02.139387+0000","last_became_active":"2023-10-14T11:59:02.139117+0000","last_became_peered":"2023-10-14T11:59:02.139117+0000","last_unstale":"2023-10-14T11:59:02.139387+0000","last_undegraded":"2023-10-14T11:59:02.139387+0000","last_fullsized":"2023-10-14T11:59:02.139387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:27:03.970871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.138469+0000","last_change":"2023-10-14T11:59:01.179436+0000","last_active":"2023-10-14T11:59:02.138469+0000","last_peered":"2023-10-14T11:59:02.138469+0000","last_clean":"2023-10-14T11:59:02.138469+0000","last_became_active":"2023-10-14T11:59:01.179109+0000","last_became_peered":"2023-10-14T11:59:01.179109+0000","last_unstale":"2023-10-14T11:59:02.138469+0000","last_undegraded":"2023-10-14T11:59:02.138469+0000","last_fullsized":"2023-10-14T11:59:02.138469+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:11:17.801821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":3,"reported_epoch":31,"state":"creating+activating","last_fresh":"2023-10-14T11:59:02.143298+0000","last_change":"2023-10-14T11:59:02.143298+0000","last_active":"2023-10-14T11:59:01.155233+0000","last_peered":"2023-10-14T11:59:01.155233+0000","last_clean":"2023-10-14T11:59:01.155233+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T11:59:02.143298+0000","last_undegraded":"2023-10-14T11:59:02.143298+0000","last_fullsized":"2023-10-14T11:59:02.143298+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:13:03.743718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.131329+0000","last_change":"2023-10-14T11:59:01.169175+0000","last_active":"2023-10-14T11:59:02.131329+0000","last_peered":"2023-10-14T11:59:02.131329+0000","last_clean":"2023-10-14T11:59:02.131329+0000","last_became_active":"2023-10-14T11:59:01.168873+0000","last_became_peered":"2023-10-14T11:59:01.168873+0000","last_unstale":"2023-10-14T11:59:02.131329+0000","last_undegraded":"2023-10-14T11:59:02.131329+0000","last_fullsized":"2023-10-14T11:59:02.131329+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:53:43.573010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":172,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-14T11:59:01.174640+0000","last_change":"2023-10-14T11:58:10.616726+0000","last_active":"2023-10-14T11:59:01.174640+0000","last_peered":"2023-10-14T11:59:01.174640+0000","last_clean":"2023-10-14T11:59:01.174640+0000","last_became_active":"2023-10-14T11:58:10.615628+0000","last_became_peered":"2023-10-14T11:58:10.615628+0000","last_unstale":"2023-10-14T11:59:01.174640+0000","last_undegraded":"2023-10-14T11:59:01.174640+0000","last_fullsized":"2023-10-14T11:59:01.174640+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:09.001265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:09.001265+0000","last_clean_scrub_stamp":"2023-10-14T11:58:09.001265+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:22:50.496050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65244,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574862,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":716079,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574862,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":125711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84399999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574862,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":132516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]}]},{"osd":0,"up_from":13,"seq":55834574861,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":401408,"data_stored":716079,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574861,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":125711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.254}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574862,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":125711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574862,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":125711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574862,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":125711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.311}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T11:59:10.846 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T11:59:10.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.982+0000 7fd1c3ccf700 1 Processor -- start 2023-10-14T11:59:10.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.983+0000 7fd1c3ccf700 1 -- start start 2023-10-14T11:59:10.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.983+0000 7fd1c3ccf700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd1bc0c1ab0 0x7fd1bc0a8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:10.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.983+0000 7fd1c3ccf700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd1bc1524a0 0x7fd1bc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:10.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.983+0000 7fd1c3ccf700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1bc151700 con 0x7fd1bc06d7b0 2023-10-14T11:59:10.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.983+0000 7fd1c3ccf700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1bc0a89d0 con 0x7fd1bc1524a0 2023-10-14T11:59:10.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.983+0000 7fd1c3ccf700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc155ae0 con 0x7fd1bc0c1ab0 2023-10-14T11:59:10.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.984+0000 7fd1c1a6b700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd1bc0c1ab0 0x7fd1bc0a8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:10.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.984+0000 7fd1c126a700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd1bc1524a0 0x7fd1bc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:10.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.984+0000 7fd1c1a6b700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd1bc0c1ab0 0x7fd1bc0a8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.78:46036/0 (socket says 172.21.15.78:46036) 2023-10-14T11:59:10.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.984+0000 7fd1c1a6b700 1 -- 172.21.15.78:0/1650832830 learned_addr learned my addr 172.21.15.78:0/1650832830 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:10.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.984+0000 7fd1c1a6b700 1 -- 172.21.15.78:0/1650832830 >> v2:172.21.15.195:3300/0 conn(0x7fd1bc1524a0 msgr2=0x7fd1bc06d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:10.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.984+0000 7fd1c1a6b700 1 --2- 172.21.15.78:0/1650832830 >> v2:172.21.15.195:3300/0 conn(0x7fd1bc1524a0 0x7fd1bc06d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:10.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.984+0000 7fd1c1a6b700 1 -- 172.21.15.78:0/1650832830 >> v1:172.21.15.195:6789/0 conn(0x7fd1bc06d7b0 legacy=0x7fd1bc14ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:10.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.984+0000 7fd1c1a6b700 1 -- 172.21.15.78:0/1650832830 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1bc1519c0 con 0x7fd1bc0c1ab0 2023-10-14T11:59:10.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.984+0000 7fd1c126a700 1 --2- 172.21.15.78:0/1650832830 >> v2:172.21.15.195:3300/0 conn(0x7fd1bc1524a0 0x7fd1bc06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T11:59:10.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.984+0000 7fd1c1a6b700 1 --2- 172.21.15.78:0/1650832830 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd1bc0c1ab0 0x7fd1bc0a8490 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac0098f0 tx=0x7fd1ac009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb621626fbb3bd62 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:10.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.984+0000 7fd1c0a69700 1 -- 172.21.15.78:0/1650832830 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1ac011830 con 0x7fd1bc0c1ab0 2023-10-14T11:59:10.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.984+0000 7fd1c0a69700 1 -- 172.21.15.78:0/1650832830 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1ac011990 con 0x7fd1bc0c1ab0 2023-10-14T11:59:10.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.984+0000 7fd1c0a69700 1 -- 172.21.15.78:0/1650832830 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1ac011d20 con 0x7fd1bc0c1ab0 2023-10-14T11:59:10.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.985+0000 7fd1c3ccf700 1 -- 172.21.15.78:0/1650832830 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd1bc0c1ab0 msgr2=0x7fd1bc0a8490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:10.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.985+0000 7fd1c3ccf700 1 --2- 172.21.15.78:0/1650832830 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd1bc0c1ab0 0x7fd1bc0a8490 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac0098f0 tx=0x7fd1ac009c00 comp rx=0 tx=0).stop 2023-10-14T11:59:10.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.985+0000 7fd1c3ccf700 1 -- 172.21.15.78:0/1650832830 shutdown_connections 2023-10-14T11:59:10.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.985+0000 7fd1c3ccf700 1 --2- 172.21.15.78:0/1650832830 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd1bc0c1ab0 0x7fd1bc0a8490 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:10.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.985+0000 7fd1c3ccf700 1 --2- 172.21.15.78:0/1650832830 >> v2:172.21.15.195:3300/0 conn(0x7fd1bc1524a0 0x7fd1bc06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:10.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.985+0000 7fd1c3ccf700 1 -- 172.21.15.78:0/1650832830 >> 172.21.15.78:0/1650832830 conn(0x7fd1bc0c2930 msgr2=0x7fd1bc0c2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:10.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.985+0000 7fd1c3ccf700 1 -- 172.21.15.78:0/1650832830 shutdown_connections 2023-10-14T11:59:10.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.985+0000 7fd1c3ccf700 1 -- 172.21.15.78:0/1650832830 wait complete. 2023-10-14T11:59:10.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.986+0000 7fd1c3ccf700 1 Processor -- start 2023-10-14T11:59:10.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.986+0000 7fd1c3ccf700 1 -- start start 2023-10-14T11:59:10.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.987+0000 7fd1c3ccf700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd1bc06d7b0 0x7fd1bc1e8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:10.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.987+0000 7fd1c3ccf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1bc0c1ab0 0x7fd1bc0bb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:10.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.987+0000 7fd1c3ccf700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd1bc1524a0 0x7fd1bc0bd980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:10.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.987+0000 7fd1c3ccf700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc1e9230 con 0x7fd1bc0c1ab0 2023-10-14T11:59:10.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.987+0000 7fd1c3ccf700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc14c4d0 con 0x7fd1bc06d7b0 2023-10-14T11:59:10.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.987+0000 7fd1c3ccf700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc0bfe30 con 0x7fd1bc1524a0 2023-10-14T11:59:10.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.987+0000 7fd1c126a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1bc0c1ab0 0x7fd1bc0bb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:10.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.987+0000 7fd1c126a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1bc0c1ab0 0x7fd1bc0bb6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46372/0 (socket says 172.21.15.78:46372) 2023-10-14T11:59:10.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.987+0000 7fd1c126a700 1 -- 172.21.15.78:0/154210314 learned_addr learned my addr 172.21.15.78:0/154210314 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:10.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.987+0000 7fd1c226c700 1 --2- 172.21.15.78:0/154210314 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd1bc1524a0 0x7fd1bc0bd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:10.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.987+0000 7fd1c1a6b700 1 --2- 172.21.15.78:0/154210314 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd1bc06d7b0 0x7fd1bc1e8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:10.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.987+0000 7fd1c126a700 1 -- 172.21.15.78:0/154210314 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd1bc1524a0 msgr2=0x7fd1bc0bd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:10.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.987+0000 7fd1c126a700 1 --2- 172.21.15.78:0/154210314 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd1bc1524a0 0x7fd1bc0bd980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:10.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.987+0000 7fd1c126a700 1 -- 172.21.15.78:0/154210314 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd1bc06d7b0 msgr2=0x7fd1bc1e8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:10.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.987+0000 7fd1c126a700 1 --2- 172.21.15.78:0/154210314 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd1bc06d7b0 0x7fd1bc1e8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:10.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.987+0000 7fd1c126a700 1 -- 172.21.15.78:0/154210314 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b80094a0 con 0x7fd1bc0c1ab0 2023-10-14T11:59:10.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.988+0000 7fd1c126a700 1 --2- 172.21.15.78:0/154210314 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1bc0c1ab0 0x7fd1bc0bb6b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fd1b800ebe0 tx=0x7fd1b8009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:10.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.988+0000 7fd1b27fc700 1 -- 172.21.15.78:0/154210314 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b800c8b0 con 0x7fd1bc0c1ab0 2023-10-14T11:59:10.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.988+0000 7fd1b27fc700 1 -- 172.21.15.78:0/154210314 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1b800ca10 con 0x7fd1bc0c1ab0 2023-10-14T11:59:10.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.988+0000 7fd1c3ccf700 1 -- 172.21.15.78:0/154210314 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1ac009550 con 0x7fd1bc0c1ab0 2023-10-14T11:59:10.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.988+0000 7fd1b27fc700 1 -- 172.21.15.78:0/154210314 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b800cda0 con 0x7fd1bc0c1ab0 2023-10-14T11:59:11.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.988+0000 7fd1c3ccf700 1 -- 172.21.15.78:0/154210314 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1bc0c03f0 con 0x7fd1bc0c1ab0 2023-10-14T11:59:11.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.989+0000 7fd1c3ccf700 1 -- 172.21.15.78:0/154210314 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1bc0c05e0 con 0x7fd1bc0c1ab0 2023-10-14T11:59:11.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.989+0000 7fd1b27fc700 1 -- 172.21.15.78:0/154210314 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd1b8013470 con 0x7fd1bc0c1ab0 2023-10-14T11:59:11.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.990+0000 7fd1b27fc700 1 --2- 172.21.15.78:0/154210314 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd1a006af00 0x7fd1a006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:11.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.990+0000 7fd1b27fc700 1 -- 172.21.15.78:0/154210314 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7fd1b8089bd0 con 0x7fd1bc0c1ab0 2023-10-14T11:59:11.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.990+0000 7fd1c1a6b700 1 --2- 172.21.15.78:0/154210314 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd1a006af00 0x7fd1a006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:11.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.990+0000 7fd1c1a6b700 1 --2- 172.21.15.78:0/154210314 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd1a006af00 0x7fd1a006d3c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac009c40 tx=0x7fd1ac008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:11.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:10.994+0000 7fd1b27fc700 1 -- 172.21.15.78:0/154210314 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd1b8054b50 con 0x7fd1bc0c1ab0 2023-10-14T11:59:11.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.180+0000 7fd1c3ccf700 1 -- 172.21.15.78:0/154210314 --> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd1bc151f80 con 0x7fd1a006af00 2023-10-14T11:59:11.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.187+0000 7fd1b27fc700 1 -- 172.21.15.78:0/154210314 <== mgr.4104 v2:172.21.15.78:6832/3171890074 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264770 (secure 0 0 0) 0x7fd1ac011b80 con 0x7fd1a006af00 2023-10-14T11:59:11.192 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:59:11.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.190+0000 7fd1c3ccf700 1 -- 172.21.15.78:0/154210314 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd1a006af00 msgr2=0x7fd1a006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:11.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.190+0000 7fd1c3ccf700 1 --2- 172.21.15.78:0/154210314 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd1a006af00 0x7fd1a006d3c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac009c40 tx=0x7fd1ac008030 comp rx=0 tx=0).stop 2023-10-14T11:59:11.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.190+0000 7fd1c3ccf700 1 -- 172.21.15.78:0/154210314 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1bc0c1ab0 msgr2=0x7fd1bc0bb6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:11.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.190+0000 7fd1c3ccf700 1 --2- 172.21.15.78:0/154210314 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1bc0c1ab0 0x7fd1bc0bb6b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fd1b800ebe0 tx=0x7fd1b8009e20 comp rx=0 tx=0).stop 2023-10-14T11:59:11.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.190+0000 7fd1c3ccf700 1 -- 172.21.15.78:0/154210314 shutdown_connections 2023-10-14T11:59:11.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.190+0000 7fd1c3ccf700 1 --2- 172.21.15.78:0/154210314 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd1bc1524a0 0x7fd1bc0bd980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:11.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.190+0000 7fd1c3ccf700 1 --2- 172.21.15.78:0/154210314 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd1a006af00 0x7fd1a006d3c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:11.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.190+0000 7fd1c3ccf700 1 --2- 172.21.15.78:0/154210314 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd1bc06d7b0 0x7fd1bc1e8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:11.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.190+0000 7fd1c3ccf700 1 --2- 172.21.15.78:0/154210314 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1bc0c1ab0 0x7fd1bc0bb6b0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:11.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.190+0000 7fd1c3ccf700 1 -- 172.21.15.78:0/154210314 >> 172.21.15.78:0/154210314 conn(0x7fd1bc0c2930 msgr2=0x7fd1bc14def0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:11.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.190+0000 7fd1c3ccf700 1 -- 172.21.15.78:0/154210314 shutdown_connections 2023-10-14T11:59:11.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.190+0000 7fd1c3ccf700 1 -- 172.21.15.78:0/154210314 wait complete. 2023-10-14T11:59:11.196 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2023-10-14T11:59:11.203 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":57,"stamp":"2023-10-14T11:59:10.976836+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219192,"kb_used_data":3008,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755528,"statfs":{"total":773094113280,"available":772869660672,"internally_reserved":0,"allocated":3080192,"data_stored":2296284,"data_compressed":65244,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.848582"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:03.133392+0000","last_change":"2023-10-14T11:59:02.143338+0000","last_active":"2023-10-14T11:59:03.133392+0000","last_peered":"2023-10-14T11:59:03.133392+0000","last_clean":"2023-10-14T11:59:03.133392+0000","last_became_active":"2023-10-14T11:59:02.143049+0000","last_became_peered":"2023-10-14T11:59:02.143049+0000","last_unstale":"2023-10-14T11:59:03.133392+0000","last_undegraded":"2023-10-14T11:59:03.133392+0000","last_fullsized":"2023-10-14T11:59:03.133392+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:05:38.808781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.233039+0000","last_change":"2023-10-14T11:59:02.148031+0000","last_active":"2023-10-14T11:59:04.233039+0000","last_peered":"2023-10-14T11:59:04.233039+0000","last_clean":"2023-10-14T11:59:04.233039+0000","last_became_active":"2023-10-14T11:59:02.147729+0000","last_became_peered":"2023-10-14T11:59:02.147729+0000","last_unstale":"2023-10-14T11:59:04.233039+0000","last_undegraded":"2023-10-14T11:59:04.233039+0000","last_fullsized":"2023-10-14T11:59:04.233039+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:32:11.548349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:03.132275+0000","last_change":"2023-10-14T11:59:02.144594+0000","last_active":"2023-10-14T11:59:03.132275+0000","last_peered":"2023-10-14T11:59:03.132275+0000","last_clean":"2023-10-14T11:59:03.132275+0000","last_became_active":"2023-10-14T11:59:02.144322+0000","last_became_peered":"2023-10-14T11:59:02.144322+0000","last_unstale":"2023-10-14T11:59:03.132275+0000","last_undegraded":"2023-10-14T11:59:03.132275+0000","last_fullsized":"2023-10-14T11:59:03.132275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:39:09.289664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.101914+0000","last_change":"2023-10-14T11:59:02.146281+0000","last_active":"2023-10-14T11:59:04.101914+0000","last_peered":"2023-10-14T11:59:04.101914+0000","last_clean":"2023-10-14T11:59:04.101914+0000","last_became_active":"2023-10-14T11:59:02.146015+0000","last_became_peered":"2023-10-14T11:59:02.146015+0000","last_unstale":"2023-10-14T11:59:04.101914+0000","last_undegraded":"2023-10-14T11:59:04.101914+0000","last_fullsized":"2023-10-14T11:59:04.101914+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:36:52.548523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.233040+0000","last_change":"2023-10-14T11:59:02.140518+0000","last_active":"2023-10-14T11:59:04.233040+0000","last_peered":"2023-10-14T11:59:04.233040+0000","last_clean":"2023-10-14T11:59:04.233040+0000","last_became_active":"2023-10-14T11:59:02.140238+0000","last_became_peered":"2023-10-14T11:59:02.140238+0000","last_unstale":"2023-10-14T11:59:04.233040+0000","last_undegraded":"2023-10-14T11:59:04.233040+0000","last_fullsized":"2023-10-14T11:59:04.233040+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:05:58.896548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.233126+0000","last_change":"2023-10-14T11:59:02.143900+0000","last_active":"2023-10-14T11:59:04.233126+0000","last_peered":"2023-10-14T11:59:04.233126+0000","last_clean":"2023-10-14T11:59:04.233126+0000","last_became_active":"2023-10-14T11:59:02.143608+0000","last_became_peered":"2023-10-14T11:59:02.143608+0000","last_unstale":"2023-10-14T11:59:04.233126+0000","last_undegraded":"2023-10-14T11:59:04.233126+0000","last_fullsized":"2023-10-14T11:59:04.233126+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:54:12.013262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.232468+0000","last_change":"2023-10-14T11:59:02.148106+0000","last_active":"2023-10-14T11:59:04.232468+0000","last_peered":"2023-10-14T11:59:04.232468+0000","last_clean":"2023-10-14T11:59:04.232468+0000","last_became_active":"2023-10-14T11:59:02.147666+0000","last_became_peered":"2023-10-14T11:59:02.147666+0000","last_unstale":"2023-10-14T11:59:04.232468+0000","last_undegraded":"2023-10-14T11:59:04.232468+0000","last_fullsized":"2023-10-14T11:59:04.232468+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:35:59.785182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.232362+0000","last_change":"2023-10-14T11:59:02.149151+0000","last_active":"2023-10-14T11:59:04.232362+0000","last_peered":"2023-10-14T11:59:04.232362+0000","last_clean":"2023-10-14T11:59:04.232362+0000","last_became_active":"2023-10-14T11:59:02.148874+0000","last_became_peered":"2023-10-14T11:59:02.148874+0000","last_unstale":"2023-10-14T11:59:04.232362+0000","last_undegraded":"2023-10-14T11:59:04.232362+0000","last_fullsized":"2023-10-14T11:59:04.232362+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:11:18.803472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:03.132911+0000","last_change":"2023-10-14T11:59:02.144928+0000","last_active":"2023-10-14T11:59:03.132911+0000","last_peered":"2023-10-14T11:59:03.132911+0000","last_clean":"2023-10-14T11:59:03.132911+0000","last_became_active":"2023-10-14T11:59:02.144644+0000","last_became_peered":"2023-10-14T11:59:02.144644+0000","last_unstale":"2023-10-14T11:59:03.132911+0000","last_undegraded":"2023-10-14T11:59:03.132911+0000","last_fullsized":"2023-10-14T11:59:03.132911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:12:09.205507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.102204+0000","last_change":"2023-10-14T11:59:02.148801+0000","last_active":"2023-10-14T11:59:04.102204+0000","last_peered":"2023-10-14T11:59:04.102204+0000","last_clean":"2023-10-14T11:59:04.102204+0000","last_became_active":"2023-10-14T11:59:02.148360+0000","last_became_peered":"2023-10-14T11:59:02.148360+0000","last_unstale":"2023-10-14T11:59:04.102204+0000","last_undegraded":"2023-10-14T11:59:04.102204+0000","last_fullsized":"2023-10-14T11:59:04.102204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:07:15.007617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.232656+0000","last_change":"2023-10-14T11:59:02.146276+0000","last_active":"2023-10-14T11:59:04.232656+0000","last_peered":"2023-10-14T11:59:04.232656+0000","last_clean":"2023-10-14T11:59:04.232656+0000","last_became_active":"2023-10-14T11:59:02.145978+0000","last_became_peered":"2023-10-14T11:59:02.145978+0000","last_unstale":"2023-10-14T11:59:04.232656+0000","last_undegraded":"2023-10-14T11:59:04.232656+0000","last_fullsized":"2023-10-14T11:59:04.232656+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:41:25.506933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:03.131770+0000","last_change":"2023-10-14T11:59:02.139674+0000","last_active":"2023-10-14T11:59:03.131770+0000","last_peered":"2023-10-14T11:59:03.131770+0000","last_clean":"2023-10-14T11:59:03.131770+0000","last_became_active":"2023-10-14T11:59:02.139404+0000","last_became_peered":"2023-10-14T11:59:02.139404+0000","last_unstale":"2023-10-14T11:59:03.131770+0000","last_undegraded":"2023-10-14T11:59:03.131770+0000","last_fullsized":"2023-10-14T11:59:03.131770+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:41:25.506933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.233207+0000","last_change":"2023-10-14T11:59:02.145057+0000","last_active":"2023-10-14T11:59:04.233207+0000","last_peered":"2023-10-14T11:59:04.233207+0000","last_clean":"2023-10-14T11:59:04.233207+0000","last_became_active":"2023-10-14T11:59:02.144776+0000","last_became_peered":"2023-10-14T11:59:02.144776+0000","last_unstale":"2023-10-14T11:59:04.233207+0000","last_undegraded":"2023-10-14T11:59:04.233207+0000","last_fullsized":"2023-10-14T11:59:04.233207+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:15:00.593091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.232748+0000","last_change":"2023-10-14T11:59:02.149459+0000","last_active":"2023-10-14T11:59:04.232748+0000","last_peered":"2023-10-14T11:59:04.232748+0000","last_clean":"2023-10-14T11:59:04.232748+0000","last_became_active":"2023-10-14T11:59:02.149184+0000","last_became_peered":"2023-10-14T11:59:02.149184+0000","last_unstale":"2023-10-14T11:59:04.232748+0000","last_undegraded":"2023-10-14T11:59:04.232748+0000","last_fullsized":"2023-10-14T11:59:04.232748+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:25:44.193754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:03.131531+0000","last_change":"2023-10-14T11:59:02.134707+0000","last_active":"2023-10-14T11:59:03.131531+0000","last_peered":"2023-10-14T11:59:03.131531+0000","last_clean":"2023-10-14T11:59:03.131531+0000","last_became_active":"2023-10-14T11:59:02.134424+0000","last_became_peered":"2023-10-14T11:59:02.134424+0000","last_unstale":"2023-10-14T11:59:03.131531+0000","last_undegraded":"2023-10-14T11:59:03.131531+0000","last_fullsized":"2023-10-14T11:59:03.131531+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:52:32.382298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.232564+0000","last_change":"2023-10-14T11:59:02.150914+0000","last_active":"2023-10-14T11:59:04.232564+0000","last_peered":"2023-10-14T11:59:04.232564+0000","last_clean":"2023-10-14T11:59:04.232564+0000","last_became_active":"2023-10-14T11:59:02.150624+0000","last_became_peered":"2023-10-14T11:59:02.150624+0000","last_unstale":"2023-10-14T11:59:04.232564+0000","last_undegraded":"2023-10-14T11:59:04.232564+0000","last_fullsized":"2023-10-14T11:59:04.232564+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:25:44.193754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.232691+0000","last_change":"2023-10-14T11:59:02.148764+0000","last_active":"2023-10-14T11:59:04.232691+0000","last_peered":"2023-10-14T11:59:04.232691+0000","last_clean":"2023-10-14T11:59:04.232691+0000","last_became_active":"2023-10-14T11:59:02.148450+0000","last_became_peered":"2023-10-14T11:59:02.148450+0000","last_unstale":"2023-10-14T11:59:04.232691+0000","last_undegraded":"2023-10-14T11:59:04.232691+0000","last_fullsized":"2023-10-14T11:59:04.232691+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:35:18.631249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.233243+0000","last_change":"2023-10-14T11:59:02.143669+0000","last_active":"2023-10-14T11:59:04.233243+0000","last_peered":"2023-10-14T11:59:04.233243+0000","last_clean":"2023-10-14T11:59:04.233243+0000","last_became_active":"2023-10-14T11:59:02.143403+0000","last_became_peered":"2023-10-14T11:59:02.143403+0000","last_unstale":"2023-10-14T11:59:04.233243+0000","last_undegraded":"2023-10-14T11:59:04.233243+0000","last_fullsized":"2023-10-14T11:59:04.233243+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:30:02.050677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.232789+0000","last_change":"2023-10-14T11:59:02.141545+0000","last_active":"2023-10-14T11:59:04.232789+0000","last_peered":"2023-10-14T11:59:04.232789+0000","last_clean":"2023-10-14T11:59:04.232789+0000","last_became_active":"2023-10-14T11:59:02.141281+0000","last_became_peered":"2023-10-14T11:59:02.141281+0000","last_unstale":"2023-10-14T11:59:04.232789+0000","last_undegraded":"2023-10-14T11:59:04.232789+0000","last_fullsized":"2023-10-14T11:59:04.232789+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:35:18.631249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.233305+0000","last_change":"2023-10-14T11:59:02.145736+0000","last_active":"2023-10-14T11:59:04.233305+0000","last_peered":"2023-10-14T11:59:04.233305+0000","last_clean":"2023-10-14T11:59:04.233305+0000","last_became_active":"2023-10-14T11:59:02.145465+0000","last_became_peered":"2023-10-14T11:59:02.145465+0000","last_unstale":"2023-10-14T11:59:04.233305+0000","last_undegraded":"2023-10-14T11:59:04.233305+0000","last_fullsized":"2023-10-14T11:59:04.233305+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:39:49.443148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:03.132303+0000","last_change":"2023-10-14T11:59:02.144592+0000","last_active":"2023-10-14T11:59:03.132303+0000","last_peered":"2023-10-14T11:59:03.132303+0000","last_clean":"2023-10-14T11:59:03.132303+0000","last_became_active":"2023-10-14T11:59:02.144310+0000","last_became_peered":"2023-10-14T11:59:02.144310+0000","last_unstale":"2023-10-14T11:59:03.132303+0000","last_undegraded":"2023-10-14T11:59:03.132303+0000","last_fullsized":"2023-10-14T11:59:03.132303+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:36:52.548523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:03.132432+0000","last_change":"2023-10-14T11:59:02.151368+0000","last_active":"2023-10-14T11:59:03.132432+0000","last_peered":"2023-10-14T11:59:03.132432+0000","last_clean":"2023-10-14T11:59:03.132432+0000","last_became_active":"2023-10-14T11:59:02.151087+0000","last_became_peered":"2023-10-14T11:59:02.151087+0000","last_unstale":"2023-10-14T11:59:03.132432+0000","last_undegraded":"2023-10-14T11:59:03.132432+0000","last_fullsized":"2023-10-14T11:59:03.132432+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:58:07.847931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.233724+0000","last_change":"2023-10-14T11:59:02.147582+0000","last_active":"2023-10-14T11:59:04.233724+0000","last_peered":"2023-10-14T11:59:04.233724+0000","last_clean":"2023-10-14T11:59:04.233724+0000","last_became_active":"2023-10-14T11:59:02.147307+0000","last_became_peered":"2023-10-14T11:59:02.147307+0000","last_unstale":"2023-10-14T11:59:04.233724+0000","last_undegraded":"2023-10-14T11:59:04.233724+0000","last_fullsized":"2023-10-14T11:59:04.233724+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:14.681110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:03.131357+0000","last_change":"2023-10-14T11:59:02.149221+0000","last_active":"2023-10-14T11:59:03.131357+0000","last_peered":"2023-10-14T11:59:03.131357+0000","last_clean":"2023-10-14T11:59:03.131357+0000","last_became_active":"2023-10-14T11:59:02.148952+0000","last_became_peered":"2023-10-14T11:59:02.148952+0000","last_unstale":"2023-10-14T11:59:03.131357+0000","last_undegraded":"2023-10-14T11:59:03.131357+0000","last_fullsized":"2023-10-14T11:59:03.131357+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:51:48.872982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.102332+0000","last_change":"2023-10-14T11:59:02.146060+0000","last_active":"2023-10-14T11:59:04.102332+0000","last_peered":"2023-10-14T11:59:04.102332+0000","last_clean":"2023-10-14T11:59:04.102332+0000","last_became_active":"2023-10-14T11:59:02.145805+0000","last_became_peered":"2023-10-14T11:59:02.145805+0000","last_unstale":"2023-10-14T11:59:04.102332+0000","last_undegraded":"2023-10-14T11:59:04.102332+0000","last_fullsized":"2023-10-14T11:59:04.102332+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:35:59.785182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.233298+0000","last_change":"2023-10-14T11:59:02.150461+0000","last_active":"2023-10-14T11:59:04.233298+0000","last_peered":"2023-10-14T11:59:04.233298+0000","last_clean":"2023-10-14T11:59:04.233298+0000","last_became_active":"2023-10-14T11:59:02.150217+0000","last_became_peered":"2023-10-14T11:59:02.150217+0000","last_unstale":"2023-10-14T11:59:04.233298+0000","last_undegraded":"2023-10-14T11:59:04.233298+0000","last_fullsized":"2023-10-14T11:59:04.233298+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:39:09.289664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.102213+0000","last_change":"2023-10-14T11:59:02.146857+0000","last_active":"2023-10-14T11:59:04.102213+0000","last_peered":"2023-10-14T11:59:04.102213+0000","last_clean":"2023-10-14T11:59:04.102213+0000","last_became_active":"2023-10-14T11:59:02.146627+0000","last_became_peered":"2023-10-14T11:59:02.146627+0000","last_unstale":"2023-10-14T11:59:04.102213+0000","last_undegraded":"2023-10-14T11:59:04.102213+0000","last_fullsized":"2023-10-14T11:59:04.102213+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:11:18.803472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:03.131914+0000","last_change":"2023-10-14T11:59:02.143747+0000","last_active":"2023-10-14T11:59:03.131914+0000","last_peered":"2023-10-14T11:59:03.131914+0000","last_clean":"2023-10-14T11:59:03.131914+0000","last_became_active":"2023-10-14T11:59:02.143421+0000","last_became_peered":"2023-10-14T11:59:02.143421+0000","last_unstale":"2023-10-14T11:59:03.131914+0000","last_undegraded":"2023-10-14T11:59:03.131914+0000","last_fullsized":"2023-10-14T11:59:03.131914+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:32:11.548349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:03.132285+0000","last_change":"2023-10-14T11:59:02.139034+0000","last_active":"2023-10-14T11:59:03.132285+0000","last_peered":"2023-10-14T11:59:03.132285+0000","last_clean":"2023-10-14T11:59:03.132285+0000","last_became_active":"2023-10-14T11:59:02.138730+0000","last_became_peered":"2023-10-14T11:59:02.138730+0000","last_unstale":"2023-10-14T11:59:03.132285+0000","last_undegraded":"2023-10-14T11:59:03.132285+0000","last_fullsized":"2023-10-14T11:59:03.132285+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:14.681110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.232515+0000","last_change":"2023-10-14T11:59:02.147709+0000","last_active":"2023-10-14T11:59:04.232515+0000","last_peered":"2023-10-14T11:59:04.232515+0000","last_clean":"2023-10-14T11:59:04.232515+0000","last_became_active":"2023-10-14T11:59:02.147385+0000","last_became_peered":"2023-10-14T11:59:02.147385+0000","last_unstale":"2023-10-14T11:59:04.232515+0000","last_undegraded":"2023-10-14T11:59:04.232515+0000","last_fullsized":"2023-10-14T11:59:04.232515+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:51:48.872982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:03.133397+0000","last_change":"2023-10-14T11:59:02.147844+0000","last_active":"2023-10-14T11:59:03.133397+0000","last_peered":"2023-10-14T11:59:03.133397+0000","last_clean":"2023-10-14T11:59:03.133397+0000","last_became_active":"2023-10-14T11:59:02.147583+0000","last_became_peered":"2023-10-14T11:59:02.147583+0000","last_unstale":"2023-10-14T11:59:03.133397+0000","last_undegraded":"2023-10-14T11:59:03.133397+0000","last_fullsized":"2023-10-14T11:59:03.133397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:47:13.361893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.232816+0000","last_change":"2023-10-14T11:59:02.146925+0000","last_active":"2023-10-14T11:59:04.232816+0000","last_peered":"2023-10-14T11:59:04.232816+0000","last_clean":"2023-10-14T11:59:04.232816+0000","last_became_active":"2023-10-14T11:59:02.146551+0000","last_became_peered":"2023-10-14T11:59:02.146551+0000","last_unstale":"2023-10-14T11:59:04.232816+0000","last_undegraded":"2023-10-14T11:59:04.232816+0000","last_fullsized":"2023-10-14T11:59:04.232816+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:54:12.013262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.233858+0000","last_change":"2023-10-14T11:59:02.147684+0000","last_active":"2023-10-14T11:59:04.233858+0000","last_peered":"2023-10-14T11:59:04.233858+0000","last_clean":"2023-10-14T11:59:04.233858+0000","last_became_active":"2023-10-14T11:59:02.147284+0000","last_became_peered":"2023-10-14T11:59:02.147284+0000","last_unstale":"2023-10-14T11:59:04.233858+0000","last_undegraded":"2023-10-14T11:59:04.233858+0000","last_fullsized":"2023-10-14T11:59:04.233858+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:30:41.965448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"31'4","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.194541+0000","last_change":"2023-10-14T11:59:01.170225+0000","last_active":"2023-10-14T11:59:02.194541+0000","last_peered":"2023-10-14T11:59:02.194541+0000","last_clean":"2023-10-14T11:59:02.194541+0000","last_became_active":"2023-10-14T11:59:01.169927+0000","last_became_peered":"2023-10-14T11:59:01.169927+0000","last_unstale":"2023-10-14T11:59:02.194541+0000","last_undegraded":"2023-10-14T11:59:02.194541+0000","last_fullsized":"2023-10-14T11:59:02.194541+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:25.623744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:03.133571+0000","last_change":"2023-10-14T11:59:02.139277+0000","last_active":"2023-10-14T11:59:03.133571+0000","last_peered":"2023-10-14T11:59:03.133571+0000","last_clean":"2023-10-14T11:59:03.133571+0000","last_became_active":"2023-10-14T11:59:02.138985+0000","last_became_peered":"2023-10-14T11:59:02.138985+0000","last_unstale":"2023-10-14T11:59:03.133571+0000","last_undegraded":"2023-10-14T11:59:03.133571+0000","last_fullsized":"2023-10-14T11:59:03.133571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:57:18.507681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"31'1","reported_seq":13,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.196067+0000","last_change":"2023-10-14T11:59:01.178406+0000","last_active":"2023-10-14T11:59:02.196067+0000","last_peered":"2023-10-14T11:59:02.196067+0000","last_clean":"2023-10-14T11:59:02.196067+0000","last_became_active":"2023-10-14T11:59:01.178136+0000","last_became_peered":"2023-10-14T11:59:01.178136+0000","last_unstale":"2023-10-14T11:59:02.196067+0000","last_undegraded":"2023-10-14T11:59:02.196067+0000","last_fullsized":"2023-10-14T11:59:02.196067+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:43:52.203439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:03.132652+0000","last_change":"2023-10-14T11:59:02.139738+0000","last_active":"2023-10-14T11:59:03.132652+0000","last_peered":"2023-10-14T11:59:03.132652+0000","last_clean":"2023-10-14T11:59:03.132652+0000","last_became_active":"2023-10-14T11:59:02.139431+0000","last_became_peered":"2023-10-14T11:59:02.139431+0000","last_unstale":"2023-10-14T11:59:03.132652+0000","last_undegraded":"2023-10-14T11:59:03.132652+0000","last_fullsized":"2023-10-14T11:59:03.132652+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:08:25.866669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"31'2","reported_seq":14,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.196490+0000","last_change":"2023-10-14T11:59:01.179865+0000","last_active":"2023-10-14T11:59:02.196490+0000","last_peered":"2023-10-14T11:59:02.196490+0000","last_clean":"2023-10-14T11:59:02.196490+0000","last_became_active":"2023-10-14T11:59:01.178404+0000","last_became_peered":"2023-10-14T11:59:01.178404+0000","last_unstale":"2023-10-14T11:59:02.196490+0000","last_undegraded":"2023-10-14T11:59:02.196490+0000","last_fullsized":"2023-10-14T11:59:02.196490+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:30:40.963797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.102422+0000","last_change":"2023-10-14T11:59:02.146195+0000","last_active":"2023-10-14T11:59:04.102422+0000","last_peered":"2023-10-14T11:59:04.102422+0000","last_clean":"2023-10-14T11:59:04.102422+0000","last_became_active":"2023-10-14T11:59:02.144184+0000","last_became_peered":"2023-10-14T11:59:02.144184+0000","last_unstale":"2023-10-14T11:59:04.102422+0000","last_undegraded":"2023-10-14T11:59:04.102422+0000","last_fullsized":"2023-10-14T11:59:04.102422+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:51:48.872982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"31'2","reported_seq":14,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.196348+0000","last_change":"2023-10-14T11:59:01.185486+0000","last_active":"2023-10-14T11:59:02.196348+0000","last_peered":"2023-10-14T11:59:02.196348+0000","last_clean":"2023-10-14T11:59:02.196348+0000","last_became_active":"2023-10-14T11:59:01.185200+0000","last_became_peered":"2023-10-14T11:59:01.185200+0000","last_unstale":"2023-10-14T11:59:02.196348+0000","last_undegraded":"2023-10-14T11:59:02.196348+0000","last_fullsized":"2023-10-14T11:59:02.196348+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:48:36.787554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.232530+0000","last_change":"2023-10-14T11:59:02.146716+0000","last_active":"2023-10-14T11:59:04.232530+0000","last_peered":"2023-10-14T11:59:04.232530+0000","last_clean":"2023-10-14T11:59:04.232530+0000","last_became_active":"2023-10-14T11:59:02.146442+0000","last_became_peered":"2023-10-14T11:59:02.146442+0000","last_unstale":"2023-10-14T11:59:04.232530+0000","last_undegraded":"2023-10-14T11:59:04.232530+0000","last_fullsized":"2023-10-14T11:59:04.232530+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:52:32.382298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"31'1","reported_seq":13,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.196263+0000","last_change":"2023-10-14T11:59:01.178774+0000","last_active":"2023-10-14T11:59:02.196263+0000","last_peered":"2023-10-14T11:59:02.196263+0000","last_clean":"2023-10-14T11:59:02.196263+0000","last_became_active":"2023-10-14T11:59:01.178378+0000","last_became_peered":"2023-10-14T11:59:01.178378+0000","last_unstale":"2023-10-14T11:59:02.196263+0000","last_undegraded":"2023-10-14T11:59:02.196263+0000","last_fullsized":"2023-10-14T11:59:02.196263+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:25.623744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:03.131682+0000","last_change":"2023-10-14T11:59:02.143614+0000","last_active":"2023-10-14T11:59:03.131682+0000","last_peered":"2023-10-14T11:59:03.131682+0000","last_clean":"2023-10-14T11:59:03.131682+0000","last_became_active":"2023-10-14T11:59:02.143332+0000","last_became_peered":"2023-10-14T11:59:02.143332+0000","last_unstale":"2023-10-14T11:59:03.131682+0000","last_undegraded":"2023-10-14T11:59:03.131682+0000","last_fullsized":"2023-10-14T11:59:03.131682+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:25:44.193754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"31'1","reported_seq":13,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.194480+0000","last_change":"2023-10-14T11:59:01.178778+0000","last_active":"2023-10-14T11:59:02.194480+0000","last_peered":"2023-10-14T11:59:02.194480+0000","last_clean":"2023-10-14T11:59:02.194480+0000","last_became_active":"2023-10-14T11:59:01.178440+0000","last_became_peered":"2023-10-14T11:59:01.178440+0000","last_unstale":"2023-10-14T11:59:02.194480+0000","last_undegraded":"2023-10-14T11:59:02.194480+0000","last_fullsized":"2023-10-14T11:59:02.194480+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:18:16.224262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:03.132233+0000","last_change":"2023-10-14T11:59:02.139185+0000","last_active":"2023-10-14T11:59:03.132233+0000","last_peered":"2023-10-14T11:59:03.132233+0000","last_clean":"2023-10-14T11:59:03.132233+0000","last_became_active":"2023-10-14T11:59:02.138862+0000","last_became_peered":"2023-10-14T11:59:02.138862+0000","last_unstale":"2023-10-14T11:59:03.132233+0000","last_undegraded":"2023-10-14T11:59:03.132233+0000","last_fullsized":"2023-10-14T11:59:03.132233+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:57:18.507681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.142184+0000","last_change":"2023-10-14T11:59:01.175662+0000","last_active":"2023-10-14T11:59:02.142184+0000","last_peered":"2023-10-14T11:59:02.142184+0000","last_clean":"2023-10-14T11:59:02.142184+0000","last_became_active":"2023-10-14T11:59:01.175405+0000","last_became_peered":"2023-10-14T11:59:01.175405+0000","last_unstale":"2023-10-14T11:59:02.142184+0000","last_undegraded":"2023-10-14T11:59:02.142184+0000","last_fullsized":"2023-10-14T11:59:02.142184+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:57:17.506030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.101795+0000","last_change":"2023-10-14T11:59:02.143559+0000","last_active":"2023-10-14T11:59:04.101795+0000","last_peered":"2023-10-14T11:59:04.101795+0000","last_clean":"2023-10-14T11:59:04.101795+0000","last_became_active":"2023-10-14T11:59:02.143303+0000","last_became_peered":"2023-10-14T11:59:02.143303+0000","last_unstale":"2023-10-14T11:59:04.101795+0000","last_undegraded":"2023-10-14T11:59:04.101795+0000","last_fullsized":"2023-10-14T11:59:04.101795+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:08:25.866669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.139909+0000","last_change":"2023-10-14T11:59:01.167233+0000","last_active":"2023-10-14T11:59:02.139909+0000","last_peered":"2023-10-14T11:59:02.139909+0000","last_clean":"2023-10-14T11:59:02.139909+0000","last_became_active":"2023-10-14T11:59:01.166905+0000","last_became_peered":"2023-10-14T11:59:01.166905+0000","last_unstale":"2023-10-14T11:59:02.139909+0000","last_undegraded":"2023-10-14T11:59:02.139909+0000","last_fullsized":"2023-10-14T11:59:02.139909+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:43:52.203439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:03.132168+0000","last_change":"2023-10-14T11:59:02.150665+0000","last_active":"2023-10-14T11:59:03.132168+0000","last_peered":"2023-10-14T11:59:03.132168+0000","last_clean":"2023-10-14T11:59:03.132168+0000","last_became_active":"2023-10-14T11:59:02.150384+0000","last_became_peered":"2023-10-14T11:59:02.150384+0000","last_unstale":"2023-10-14T11:59:03.132168+0000","last_undegraded":"2023-10-14T11:59:03.132168+0000","last_fullsized":"2023-10-14T11:59:03.132168+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:30:02.050677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.142585+0000","last_change":"2023-10-14T11:59:01.179314+0000","last_active":"2023-10-14T11:59:02.142585+0000","last_peered":"2023-10-14T11:59:02.142585+0000","last_clean":"2023-10-14T11:59:02.142585+0000","last_became_active":"2023-10-14T11:59:01.179019+0000","last_became_peered":"2023-10-14T11:59:01.179019+0000","last_unstale":"2023-10-14T11:59:02.142585+0000","last_undegraded":"2023-10-14T11:59:02.142585+0000","last_fullsized":"2023-10-14T11:59:02.142585+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:27:06.786668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.232789+0000","last_change":"2023-10-14T11:59:02.146774+0000","last_active":"2023-10-14T11:59:04.232789+0000","last_peered":"2023-10-14T11:59:04.232789+0000","last_clean":"2023-10-14T11:59:04.232789+0000","last_became_active":"2023-10-14T11:59:02.146521+0000","last_became_peered":"2023-10-14T11:59:02.146521+0000","last_unstale":"2023-10-14T11:59:04.232789+0000","last_undegraded":"2023-10-14T11:59:04.232789+0000","last_fullsized":"2023-10-14T11:59:04.232789+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:15:13.383907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.141994+0000","last_change":"2023-10-14T11:59:01.177873+0000","last_active":"2023-10-14T11:59:02.141994+0000","last_peered":"2023-10-14T11:59:02.141994+0000","last_clean":"2023-10-14T11:59:02.141994+0000","last_became_active":"2023-10-14T11:59:01.177594+0000","last_became_peered":"2023-10-14T11:59:01.177594+0000","last_unstale":"2023-10-14T11:59:02.141994+0000","last_undegraded":"2023-10-14T11:59:02.141994+0000","last_fullsized":"2023-10-14T11:59:02.141994+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:25.623744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.232690+0000","last_change":"2023-10-14T11:59:02.150092+0000","last_active":"2023-10-14T11:59:04.232690+0000","last_peered":"2023-10-14T11:59:04.232690+0000","last_clean":"2023-10-14T11:59:04.232690+0000","last_became_active":"2023-10-14T11:59:02.149799+0000","last_became_peered":"2023-10-14T11:59:02.149799+0000","last_unstale":"2023-10-14T11:59:04.232690+0000","last_undegraded":"2023-10-14T11:59:04.232690+0000","last_fullsized":"2023-10-14T11:59:04.232690+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:41:25.506933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"31'3","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.194427+0000","last_change":"2023-10-14T11:59:01.178143+0000","last_active":"2023-10-14T11:59:02.194427+0000","last_peered":"2023-10-14T11:59:02.194427+0000","last_clean":"2023-10-14T11:59:02.194427+0000","last_became_active":"2023-10-14T11:59:01.177736+0000","last_became_peered":"2023-10-14T11:59:01.177736+0000","last_unstale":"2023-10-14T11:59:02.194427+0000","last_undegraded":"2023-10-14T11:59:02.194427+0000","last_fullsized":"2023-10-14T11:59:02.194427+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:35:58.783531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:03.132409+0000","last_change":"2023-10-14T11:59:02.145816+0000","last_active":"2023-10-14T11:59:03.132409+0000","last_peered":"2023-10-14T11:59:03.132409+0000","last_clean":"2023-10-14T11:59:03.132409+0000","last_became_active":"2023-10-14T11:59:02.145548+0000","last_became_peered":"2023-10-14T11:59:02.145548+0000","last_unstale":"2023-10-14T11:59:03.132409+0000","last_undegraded":"2023-10-14T11:59:03.132409+0000","last_fullsized":"2023-10-14T11:59:03.132409+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:15:13.383907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"31'6","reported_seq":18,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.198432+0000","last_change":"2023-10-14T11:59:01.180187+0000","last_active":"2023-10-14T11:59:02.198432+0000","last_peered":"2023-10-14T11:59:02.198432+0000","last_clean":"2023-10-14T11:59:02.198432+0000","last_became_active":"2023-10-14T11:59:01.179909+0000","last_became_peered":"2023-10-14T11:59:01.179909+0000","last_unstale":"2023-10-14T11:59:02.198432+0000","last_undegraded":"2023-10-14T11:59:02.198432+0000","last_fullsized":"2023-10-14T11:59:02.198432+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:58:06.846280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.101844+0000","last_change":"2023-10-14T11:59:02.143696+0000","last_active":"2023-10-14T11:59:04.101844+0000","last_peered":"2023-10-14T11:59:04.101844+0000","last_clean":"2023-10-14T11:59:04.101844+0000","last_became_active":"2023-10-14T11:59:02.143443+0000","last_became_peered":"2023-10-14T11:59:02.143443+0000","last_unstale":"2023-10-14T11:59:04.101844+0000","last_undegraded":"2023-10-14T11:59:04.101844+0000","last_fullsized":"2023-10-14T11:59:04.101844+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:12:09.205507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"31'2","reported_seq":14,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.195220+0000","last_change":"2023-10-14T11:59:01.165901+0000","last_active":"2023-10-14T11:59:02.195220+0000","last_peered":"2023-10-14T11:59:02.195220+0000","last_clean":"2023-10-14T11:59:02.195220+0000","last_became_active":"2023-10-14T11:59:01.165597+0000","last_became_peered":"2023-10-14T11:59:01.165597+0000","last_unstale":"2023-10-14T11:59:02.195220+0000","last_undegraded":"2023-10-14T11:59:02.195220+0000","last_fullsized":"2023-10-14T11:59:02.195220+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:05:37.807130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.232620+0000","last_change":"2023-10-14T11:59:02.141427+0000","last_active":"2023-10-14T11:59:04.232620+0000","last_peered":"2023-10-14T11:59:04.232620+0000","last_clean":"2023-10-14T11:59:04.232620+0000","last_became_active":"2023-10-14T11:59:02.141166+0000","last_became_peered":"2023-10-14T11:59:02.141166+0000","last_unstale":"2023-10-14T11:59:04.232620+0000","last_undegraded":"2023-10-14T11:59:04.232620+0000","last_fullsized":"2023-10-14T11:59:04.232620+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:52:32.382298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.147442+0000","last_change":"2023-10-14T11:59:01.172037+0000","last_active":"2023-10-14T11:59:02.147442+0000","last_peered":"2023-10-14T11:59:02.147442+0000","last_clean":"2023-10-14T11:59:02.147442+0000","last_became_active":"2023-10-14T11:59:01.170760+0000","last_became_peered":"2023-10-14T11:59:01.170760+0000","last_unstale":"2023-10-14T11:59:02.147442+0000","last_undegraded":"2023-10-14T11:59:02.147442+0000","last_fullsized":"2023-10-14T11:59:02.147442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:57:17.506030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.232831+0000","last_change":"2023-10-14T11:59:02.147409+0000","last_active":"2023-10-14T11:59:04.232831+0000","last_peered":"2023-10-14T11:59:04.232831+0000","last_clean":"2023-10-14T11:59:04.232831+0000","last_became_active":"2023-10-14T11:59:02.146912+0000","last_became_peered":"2023-10-14T11:59:02.146912+0000","last_unstale":"2023-10-14T11:59:04.232831+0000","last_undegraded":"2023-10-14T11:59:04.232831+0000","last_fullsized":"2023-10-14T11:59:04.232831+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:39:09.289664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.142203+0000","last_change":"2023-10-14T11:59:01.178960+0000","last_active":"2023-10-14T11:59:02.142203+0000","last_peered":"2023-10-14T11:59:02.142203+0000","last_clean":"2023-10-14T11:59:02.142203+0000","last_became_active":"2023-10-14T11:59:01.178691+0000","last_became_peered":"2023-10-14T11:59:01.178691+0000","last_unstale":"2023-10-14T11:59:02.142203+0000","last_undegraded":"2023-10-14T11:59:02.142203+0000","last_fullsized":"2023-10-14T11:59:02.142203+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:13.679459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:03.133303+0000","last_change":"2023-10-14T11:59:02.140181+0000","last_active":"2023-10-14T11:59:03.133303+0000","last_peered":"2023-10-14T11:59:03.133303+0000","last_clean":"2023-10-14T11:59:03.133303+0000","last_became_active":"2023-10-14T11:59:02.139382+0000","last_became_peered":"2023-10-14T11:59:02.139382+0000","last_unstale":"2023-10-14T11:59:03.133303+0000","last_undegraded":"2023-10-14T11:59:03.133303+0000","last_fullsized":"2023-10-14T11:59:03.133303+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:54:28.849250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"31'8","reported_seq":20,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.197139+0000","last_change":"2023-10-14T11:59:01.178769+0000","last_active":"2023-10-14T11:59:02.197139+0000","last_peered":"2023-10-14T11:59:02.197139+0000","last_clean":"2023-10-14T11:59:02.197139+0000","last_became_active":"2023-10-14T11:59:01.178288+0000","last_became_peered":"2023-10-14T11:59:01.178288+0000","last_unstale":"2023-10-14T11:59:02.197139+0000","last_undegraded":"2023-10-14T11:59:02.197139+0000","last_fullsized":"2023-10-14T11:59:02.197139+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:07:14.005966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:03.132635+0000","last_change":"2023-10-14T11:59:02.141617+0000","last_active":"2023-10-14T11:59:03.132635+0000","last_peered":"2023-10-14T11:59:03.132635+0000","last_clean":"2023-10-14T11:59:03.132635+0000","last_became_active":"2023-10-14T11:59:02.141346+0000","last_became_peered":"2023-10-14T11:59:02.141346+0000","last_unstale":"2023-10-14T11:59:03.132635+0000","last_undegraded":"2023-10-14T11:59:03.132635+0000","last_fullsized":"2023-10-14T11:59:03.132635+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:27:07.788319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"31'1","reported_seq":13,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.196943+0000","last_change":"2023-10-14T11:59:01.183630+0000","last_active":"2023-10-14T11:59:02.196943+0000","last_peered":"2023-10-14T11:59:02.196943+0000","last_clean":"2023-10-14T11:59:02.196943+0000","last_became_active":"2023-10-14T11:59:01.183408+0000","last_became_peered":"2023-10-14T11:59:01.183408+0000","last_unstale":"2023-10-14T11:59:02.196943+0000","last_undegraded":"2023-10-14T11:59:02.196943+0000","last_fullsized":"2023-10-14T11:59:02.196943+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:12:25.623744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:03.131815+0000","last_change":"2023-10-14T11:59:02.141878+0000","last_active":"2023-10-14T11:59:03.131815+0000","last_peered":"2023-10-14T11:59:03.131815+0000","last_clean":"2023-10-14T11:59:03.131815+0000","last_became_active":"2023-10-14T11:59:02.141229+0000","last_became_peered":"2023-10-14T11:59:02.141229+0000","last_unstale":"2023-10-14T11:59:03.131815+0000","last_undegraded":"2023-10-14T11:59:03.131815+0000","last_fullsized":"2023-10-14T11:59:03.131815+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:35:18.631249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"31'2","reported_seq":14,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.193142+0000","last_change":"2023-10-14T11:59:01.165942+0000","last_active":"2023-10-14T11:59:02.193142+0000","last_peered":"2023-10-14T11:59:02.193142+0000","last_clean":"2023-10-14T11:59:02.193142+0000","last_became_active":"2023-10-14T11:59:01.165646+0000","last_became_peered":"2023-10-14T11:59:01.165646+0000","last_unstale":"2023-10-14T11:59:02.193142+0000","last_undegraded":"2023-10-14T11:59:02.193142+0000","last_fullsized":"2023-10-14T11:59:02.193142+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:14:59.591440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.102185+0000","last_change":"2023-10-14T11:59:02.147269+0000","last_active":"2023-10-14T11:59:04.102185+0000","last_peered":"2023-10-14T11:59:04.102185+0000","last_clean":"2023-10-14T11:59:04.102185+0000","last_became_active":"2023-10-14T11:59:02.147005+0000","last_became_peered":"2023-10-14T11:59:02.147005+0000","last_unstale":"2023-10-14T11:59:04.102185+0000","last_undegraded":"2023-10-14T11:59:04.102185+0000","last_fullsized":"2023-10-14T11:59:04.102185+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:13:03.743718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"31'1","reported_seq":13,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.194133+0000","last_change":"2023-10-14T11:59:01.183080+0000","last_active":"2023-10-14T11:59:02.194133+0000","last_peered":"2023-10-14T11:59:02.194133+0000","last_clean":"2023-10-14T11:59:02.194133+0000","last_became_active":"2023-10-14T11:59:01.182786+0000","last_became_peered":"2023-10-14T11:59:01.182786+0000","last_unstale":"2023-10-14T11:59:02.194133+0000","last_undegraded":"2023-10-14T11:59:02.194133+0000","last_fullsized":"2023-10-14T11:59:02.194133+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:05:27.508197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.232807+0000","last_change":"2023-10-14T11:59:02.146618+0000","last_active":"2023-10-14T11:59:04.232807+0000","last_peered":"2023-10-14T11:59:04.232807+0000","last_clean":"2023-10-14T11:59:04.232807+0000","last_became_active":"2023-10-14T11:59:02.146352+0000","last_became_peered":"2023-10-14T11:59:02.146352+0000","last_unstale":"2023-10-14T11:59:04.232807+0000","last_undegraded":"2023-10-14T11:59:04.232807+0000","last_fullsized":"2023-10-14T11:59:04.232807+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:05:28.509848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.140217+0000","last_change":"2023-10-14T11:59:01.178771+0000","last_active":"2023-10-14T11:59:02.140217+0000","last_peered":"2023-10-14T11:59:02.140217+0000","last_clean":"2023-10-14T11:59:02.140217+0000","last_became_active":"2023-10-14T11:59:01.178416+0000","last_became_peered":"2023-10-14T11:59:01.178416+0000","last_unstale":"2023-10-14T11:59:02.140217+0000","last_undegraded":"2023-10-14T11:59:02.140217+0000","last_fullsized":"2023-10-14T11:59:02.140217+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:11:46.659890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.233004+0000","last_change":"2023-10-14T11:59:02.145624+0000","last_active":"2023-10-14T11:59:04.233004+0000","last_peered":"2023-10-14T11:59:04.233004+0000","last_clean":"2023-10-14T11:59:04.233004+0000","last_became_active":"2023-10-14T11:59:02.145381+0000","last_became_peered":"2023-10-14T11:59:02.145381+0000","last_unstale":"2023-10-14T11:59:04.233004+0000","last_undegraded":"2023-10-14T11:59:04.233004+0000","last_fullsized":"2023-10-14T11:59:04.233004+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:53:44.574661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"31'2","reported_seq":14,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.196521+0000","last_change":"2023-10-14T11:59:01.177556+0000","last_active":"2023-10-14T11:59:02.196521+0000","last_peered":"2023-10-14T11:59:02.196521+0000","last_clean":"2023-10-14T11:59:02.196521+0000","last_became_active":"2023-10-14T11:59:01.177313+0000","last_became_peered":"2023-10-14T11:59:01.177313+0000","last_unstale":"2023-10-14T11:59:02.196521+0000","last_undegraded":"2023-10-14T11:59:02.196521+0000","last_fullsized":"2023-10-14T11:59:02.196521+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:27:02.969220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.233078+0000","last_change":"2023-10-14T11:59:02.141409+0000","last_active":"2023-10-14T11:59:04.233078+0000","last_peered":"2023-10-14T11:59:04.233078+0000","last_clean":"2023-10-14T11:59:04.233078+0000","last_became_active":"2023-10-14T11:59:02.141163+0000","last_became_peered":"2023-10-14T11:59:02.141163+0000","last_unstale":"2023-10-14T11:59:04.233078+0000","last_undegraded":"2023-10-14T11:59:04.233078+0000","last_fullsized":"2023-10-14T11:59:04.233078+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:20:02.525878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.143630+0000","last_change":"2023-10-14T11:59:01.168240+0000","last_active":"2023-10-14T11:59:02.143630+0000","last_peered":"2023-10-14T11:59:02.143630+0000","last_clean":"2023-10-14T11:59:02.143630+0000","last_became_active":"2023-10-14T11:59:01.167944+0000","last_became_peered":"2023-10-14T11:59:01.167944+0000","last_unstale":"2023-10-14T11:59:02.143630+0000","last_undegraded":"2023-10-14T11:59:02.143630+0000","last_fullsized":"2023-10-14T11:59:02.143630+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:27:02.969220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.232956+0000","last_change":"2023-10-14T11:59:02.148475+0000","last_active":"2023-10-14T11:59:04.232956+0000","last_peered":"2023-10-14T11:59:04.232956+0000","last_clean":"2023-10-14T11:59:04.232956+0000","last_became_active":"2023-10-14T11:59:02.148176+0000","last_became_peered":"2023-10-14T11:59:02.148176+0000","last_unstale":"2023-10-14T11:59:04.232956+0000","last_undegraded":"2023-10-14T11:59:04.232956+0000","last_fullsized":"2023-10-14T11:59:04.232956+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:11:47.661541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.139973+0000","last_change":"2023-10-14T11:59:01.176530+0000","last_active":"2023-10-14T11:59:02.139973+0000","last_peered":"2023-10-14T11:59:02.139973+0000","last_clean":"2023-10-14T11:59:02.139973+0000","last_became_active":"2023-10-14T11:59:01.176193+0000","last_became_peered":"2023-10-14T11:59:01.176193+0000","last_unstale":"2023-10-14T11:59:02.139973+0000","last_undegraded":"2023-10-14T11:59:02.139973+0000","last_fullsized":"2023-10-14T11:59:02.139973+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:54:27.847599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.232652+0000","last_change":"2023-10-14T11:59:02.148446+0000","last_active":"2023-10-14T11:59:04.232652+0000","last_peered":"2023-10-14T11:59:04.232652+0000","last_clean":"2023-10-14T11:59:04.232652+0000","last_became_active":"2023-10-14T11:59:02.147955+0000","last_became_peered":"2023-10-14T11:59:02.147955+0000","last_unstale":"2023-10-14T11:59:04.232652+0000","last_undegraded":"2023-10-14T11:59:04.232652+0000","last_fullsized":"2023-10-14T11:59:04.232652+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:32:11.548349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.156755+0000","last_change":"2023-10-14T11:59:01.178157+0000","last_active":"2023-10-14T11:59:02.156755+0000","last_peered":"2023-10-14T11:59:02.156755+0000","last_clean":"2023-10-14T11:59:02.156755+0000","last_became_active":"2023-10-14T11:59:01.177863+0000","last_became_peered":"2023-10-14T11:59:01.177863+0000","last_unstale":"2023-10-14T11:59:02.156755+0000","last_undegraded":"2023-10-14T11:59:02.156755+0000","last_fullsized":"2023-10-14T11:59:02.156755+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:57:17.506030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.232624+0000","last_change":"2023-10-14T11:59:02.143662+0000","last_active":"2023-10-14T11:59:04.232624+0000","last_peered":"2023-10-14T11:59:04.232624+0000","last_clean":"2023-10-14T11:59:04.232624+0000","last_became_active":"2023-10-14T11:59:02.143456+0000","last_became_peered":"2023-10-14T11:59:02.143456+0000","last_unstale":"2023-10-14T11:59:04.232624+0000","last_undegraded":"2023-10-14T11:59:04.232624+0000","last_fullsized":"2023-10-14T11:59:04.232624+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:41:47.404184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"31'2","reported_seq":14,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.194811+0000","last_change":"2023-10-14T11:59:01.181546+0000","last_active":"2023-10-14T11:59:02.194811+0000","last_peered":"2023-10-14T11:59:02.194811+0000","last_clean":"2023-10-14T11:59:02.194811+0000","last_became_active":"2023-10-14T11:59:01.181255+0000","last_became_peered":"2023-10-14T11:59:01.181255+0000","last_unstale":"2023-10-14T11:59:02.194811+0000","last_undegraded":"2023-10-14T11:59:02.194811+0000","last_fullsized":"2023-10-14T11:59:02.194811+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:27:02.969220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:03.132110+0000","last_change":"2023-10-14T11:59:02.142119+0000","last_active":"2023-10-14T11:59:03.132110+0000","last_peered":"2023-10-14T11:59:03.132110+0000","last_clean":"2023-10-14T11:59:03.132110+0000","last_became_active":"2023-10-14T11:59:02.141586+0000","last_became_peered":"2023-10-14T11:59:02.141586+0000","last_unstale":"2023-10-14T11:59:03.132110+0000","last_undegraded":"2023-10-14T11:59:03.132110+0000","last_fullsized":"2023-10-14T11:59:03.132110+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:54:12.013262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.234071+0000","last_change":"2023-10-14T11:59:01.180897+0000","last_active":"2023-10-14T11:59:04.234071+0000","last_peered":"2023-10-14T11:59:04.234071+0000","last_clean":"2023-10-14T11:59:04.234071+0000","last_became_active":"2023-10-14T11:59:01.180524+0000","last_became_peered":"2023-10-14T11:59:01.180524+0000","last_unstale":"2023-10-14T11:59:04.234071+0000","last_undegraded":"2023-10-14T11:59:04.234071+0000","last_fullsized":"2023-10-14T11:59:04.234071+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:18:16.224262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.233581+0000","last_change":"2023-10-14T11:59:02.144666+0000","last_active":"2023-10-14T11:59:04.233581+0000","last_peered":"2023-10-14T11:59:04.233581+0000","last_clean":"2023-10-14T11:59:04.233581+0000","last_became_active":"2023-10-14T11:59:02.144379+0000","last_became_peered":"2023-10-14T11:59:02.144379+0000","last_unstale":"2023-10-14T11:59:04.233581+0000","last_undegraded":"2023-10-14T11:59:04.233581+0000","last_fullsized":"2023-10-14T11:59:04.233581+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:57:18.507681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.147506+0000","last_change":"2023-10-14T11:59:01.183591+0000","last_active":"2023-10-14T11:59:02.147506+0000","last_peered":"2023-10-14T11:59:02.147506+0000","last_clean":"2023-10-14T11:59:02.147506+0000","last_became_active":"2023-10-14T11:59:01.183019+0000","last_became_peered":"2023-10-14T11:59:01.183019+0000","last_unstale":"2023-10-14T11:59:02.147506+0000","last_undegraded":"2023-10-14T11:59:02.147506+0000","last_fullsized":"2023-10-14T11:59:02.147506+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:05:37.807130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:03.133140+0000","last_change":"2023-10-14T11:59:02.144264+0000","last_active":"2023-10-14T11:59:03.133140+0000","last_peered":"2023-10-14T11:59:03.133140+0000","last_clean":"2023-10-14T11:59:03.133140+0000","last_became_active":"2023-10-14T11:59:02.143989+0000","last_became_peered":"2023-10-14T11:59:02.143989+0000","last_unstale":"2023-10-14T11:59:03.133140+0000","last_undegraded":"2023-10-14T11:59:03.133140+0000","last_fullsized":"2023-10-14T11:59:03.133140+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:18:17.225913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.141891+0000","last_change":"2023-10-14T11:59:01.177539+0000","last_active":"2023-10-14T11:59:02.141891+0000","last_peered":"2023-10-14T11:59:02.141891+0000","last_clean":"2023-10-14T11:59:02.141891+0000","last_became_active":"2023-10-14T11:59:01.177255+0000","last_became_peered":"2023-10-14T11:59:01.177255+0000","last_unstale":"2023-10-14T11:59:02.141891+0000","last_undegraded":"2023-10-14T11:59:02.141891+0000","last_fullsized":"2023-10-14T11:59:02.141891+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:39:48.441497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.234015+0000","last_change":"2023-10-14T11:59:02.147708+0000","last_active":"2023-10-14T11:59:04.234015+0000","last_peered":"2023-10-14T11:59:04.234015+0000","last_clean":"2023-10-14T11:59:04.234015+0000","last_became_active":"2023-10-14T11:59:02.147437+0000","last_became_peered":"2023-10-14T11:59:02.147437+0000","last_unstale":"2023-10-14T11:59:04.234015+0000","last_undegraded":"2023-10-14T11:59:04.234015+0000","last_fullsized":"2023-10-14T11:59:04.234015+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:58:07.847931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.147460+0000","last_change":"2023-10-14T11:59:01.182837+0000","last_active":"2023-10-14T11:59:02.147460+0000","last_peered":"2023-10-14T11:59:02.147460+0000","last_clean":"2023-10-14T11:59:02.147460+0000","last_became_active":"2023-10-14T11:59:01.182236+0000","last_became_peered":"2023-10-14T11:59:01.182236+0000","last_unstale":"2023-10-14T11:59:02.147460+0000","last_undegraded":"2023-10-14T11:59:02.147460+0000","last_fullsized":"2023-10-14T11:59:02.147460+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:13.679459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.232382+0000","last_change":"2023-10-14T11:59:02.148654+0000","last_active":"2023-10-14T11:59:04.232382+0000","last_peered":"2023-10-14T11:59:04.232382+0000","last_clean":"2023-10-14T11:59:04.232382+0000","last_became_active":"2023-10-14T11:59:02.148356+0000","last_became_peered":"2023-10-14T11:59:02.148356+0000","last_unstale":"2023-10-14T11:59:04.232382+0000","last_undegraded":"2023-10-14T11:59:04.232382+0000","last_fullsized":"2023-10-14T11:59:04.232382+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:51:48.872982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.139992+0000","last_change":"2023-10-14T11:59:01.168077+0000","last_active":"2023-10-14T11:59:02.139992+0000","last_peered":"2023-10-14T11:59:02.139992+0000","last_clean":"2023-10-14T11:59:02.139992+0000","last_became_active":"2023-10-14T11:59:01.167803+0000","last_became_peered":"2023-10-14T11:59:01.167803+0000","last_unstale":"2023-10-14T11:59:02.139992+0000","last_undegraded":"2023-10-14T11:59:02.139992+0000","last_fullsized":"2023-10-14T11:59:02.139992+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:47:12.360242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:03.132336+0000","last_change":"2023-10-14T11:59:02.139387+0000","last_active":"2023-10-14T11:59:03.132336+0000","last_peered":"2023-10-14T11:59:03.132336+0000","last_clean":"2023-10-14T11:59:03.132336+0000","last_became_active":"2023-10-14T11:59:02.139117+0000","last_became_peered":"2023-10-14T11:59:02.139117+0000","last_unstale":"2023-10-14T11:59:03.132336+0000","last_undegraded":"2023-10-14T11:59:03.132336+0000","last_fullsized":"2023-10-14T11:59:03.132336+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:30:41.965448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.142165+0000","last_change":"2023-10-14T11:59:01.179436+0000","last_active":"2023-10-14T11:59:02.142165+0000","last_peered":"2023-10-14T11:59:02.142165+0000","last_clean":"2023-10-14T11:59:02.142165+0000","last_became_active":"2023-10-14T11:59:01.179109+0000","last_became_peered":"2023-10-14T11:59:01.179109+0000","last_unstale":"2023-10-14T11:59:02.142165+0000","last_undegraded":"2023-10-14T11:59:02.142165+0000","last_fullsized":"2023-10-14T11:59:02.142165+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T14:11:17.801821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T11:59:04.232830+0000","last_change":"2023-10-14T11:59:02.147730+0000","last_active":"2023-10-14T11:59:04.232830+0000","last_peered":"2023-10-14T11:59:04.232830+0000","last_clean":"2023-10-14T11:59:04.232830+0000","last_became_active":"2023-10-14T11:59:02.147436+0000","last_became_peered":"2023-10-14T11:59:02.147436+0000","last_unstale":"2023-10-14T11:59:04.232830+0000","last_undegraded":"2023-10-14T11:59:04.232830+0000","last_fullsized":"2023-10-14T11:59:04.232830+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:01.155233+0000","last_clean_scrub_stamp":"2023-10-14T11:59:01.155233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T15:30:02.050677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.143648+0000","last_change":"2023-10-14T11:59:01.169175+0000","last_active":"2023-10-14T11:59:02.143648+0000","last_peered":"2023-10-14T11:59:02.143648+0000","last_clean":"2023-10-14T11:59:02.143648+0000","last_became_active":"2023-10-14T11:59:01.168873+0000","last_became_peered":"2023-10-14T11:59:01.168873+0000","last_unstale":"2023-10-14T11:59:02.143648+0000","last_undegraded":"2023-10-14T11:59:02.143648+0000","last_fullsized":"2023-10-14T11:59:02.143648+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:59:00.153582+0000","last_clean_scrub_stamp":"2023-10-14T11:59:00.153582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:53:43.573010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":173,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T11:59:02.136621+0000","last_change":"2023-10-14T11:58:10.616726+0000","last_active":"2023-10-14T11:59:02.136621+0000","last_peered":"2023-10-14T11:59:02.136621+0000","last_clean":"2023-10-14T11:59:02.136621+0000","last_became_active":"2023-10-14T11:58:10.615628+0000","last_became_peered":"2023-10-14T11:58:10.615628+0000","last_unstale":"2023-10-14T11:59:02.136621+0000","last_undegraded":"2023-10-14T11:59:02.136621+0000","last_fullsized":"2023-10-14T11:59:02.136621+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:09.001265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:09.001265+0000","last_clean_scrub_stamp":"2023-10-14T11:58:09.001265+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:22:50.496050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65244,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574863,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":430080,"data_stored":729299,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]}]},{"osd":6,"up_from":13,"seq":55834574863,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":368640,"data_stored":139457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574863,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":376832,"data_stored":140475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91900000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94099999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]}]},{"osd":0,"up_from":13,"seq":55834574862,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":729209,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574862,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":364544,"data_stored":138911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.216}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574863,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":376832,"data_stored":140475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28299999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78300000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574863,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":368640,"data_stored":138945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574863,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":368640,"data_stored":139513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T11:59:11.208 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-14T11:59:11.209 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=testcephfs1' mds allow 2023-10-14T11:59:11.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.369+0000 7f166ae7f700 1 Processor -- start 2023-10-14T11:59:11.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.369+0000 7f166ae7f700 1 -- start start 2023-10-14T11:59:11.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.370+0000 7f166ae7f700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f166413dbe0 0x7f166406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:11.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.370+0000 7f166ae7f700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f16640c4540 0x7f16640c02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:11.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.370+0000 7f166ae7f700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f166406ae10 0x7f16640c0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:11.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.370+0000 7f166ae7f700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f166406d330 con 0x7f166413dbe0 2023-10-14T11:59:11.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.370+0000 7f166ae7f700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f166406ebe0 con 0x7f166406ae10 2023-10-14T11:59:11.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.370+0000 7f166ae7f700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16640c0d60 con 0x7f16640c4540 2023-10-14T11:59:11.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.370+0000 7f1668c1b700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f166413dbe0 0x7f166406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:11.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.370+0000 7f1668c1b700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f166413dbe0 0x7f166406a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46378/0 (socket says 172.21.15.78:46378) 2023-10-14T11:59:11.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.370+0000 7f1668c1b700 1 -- 172.21.15.78:0/2373530623 learned_addr learned my addr 172.21.15.78:0/2373530623 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:11.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.371+0000 7f1668c1b700 1 -- 172.21.15.78:0/2373530623 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f16640c4540 msgr2=0x7f16640c02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:11.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.371+0000 7f1663fff700 1 --2- 172.21.15.78:0/2373530623 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f16640c4540 0x7f16640c02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:11.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.371+0000 7f1668c1b700 1 --2- 172.21.15.78:0/2373530623 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f16640c4540 0x7f16640c02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:11.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.371+0000 7f1668c1b700 1 -- 172.21.15.78:0/2373530623 >> v2:172.21.15.195:3300/0 conn(0x7f166406ae10 msgr2=0x7f16640c0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:11.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.371+0000 7f166941c700 1 --2- 172.21.15.78:0/2373530623 >> v2:172.21.15.195:3300/0 conn(0x7f166406ae10 0x7f16640c0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:11.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.371+0000 7f1668c1b700 1 --2- 172.21.15.78:0/2373530623 >> v2:172.21.15.195:3300/0 conn(0x7f166406ae10 0x7f16640c0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:11.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.371+0000 7f1668c1b700 1 -- 172.21.15.78:0/2373530623 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16641558c0 con 0x7f166413dbe0 2023-10-14T11:59:11.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.371+0000 7f166941c700 1 --2- 172.21.15.78:0/2373530623 >> v2:172.21.15.195:3300/0 conn(0x7f166406ae10 0x7f16640c0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:59:11.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.371+0000 7f1668c1b700 1 --2- 172.21.15.78:0/2373530623 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f166413dbe0 0x7f166406a8d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f166406e270 tx=0x7f1658009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bd3aa1f5b3da9cd server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:11.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.371+0000 7f16637fe700 1 -- 172.21.15.78:0/2373530623 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f165800b870 con 0x7f166413dbe0 2023-10-14T11:59:11.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.371+0000 7f16637fe700 1 -- 172.21.15.78:0/2373530623 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f165800b9d0 con 0x7f166413dbe0 2023-10-14T11:59:11.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.371+0000 7f16637fe700 1 -- 172.21.15.78:0/2373530623 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f165800bd60 con 0x7f166413dbe0 2023-10-14T11:59:11.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.371+0000 7f166ae7f700 1 -- 172.21.15.78:0/2373530623 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f166413dbe0 msgr2=0x7f166406a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:11.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.371+0000 7f166ae7f700 1 --2- 172.21.15.78:0/2373530623 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f166413dbe0 0x7f166406a8d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f166406e270 tx=0x7f1658009bd0 comp rx=0 tx=0).stop 2023-10-14T11:59:11.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.372+0000 7f166ae7f700 1 -- 172.21.15.78:0/2373530623 shutdown_connections 2023-10-14T11:59:11.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.372+0000 7f166ae7f700 1 --2- 172.21.15.78:0/2373530623 >> v2:172.21.15.195:3300/0 conn(0x7f166406ae10 0x7f16640c0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:11.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.372+0000 7f166ae7f700 1 --2- 172.21.15.78:0/2373530623 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f166413dbe0 0x7f166406a8d0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:11.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.372+0000 7f166ae7f700 1 --2- 172.21.15.78:0/2373530623 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f16640c4540 0x7f16640c02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:11.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.372+0000 7f166ae7f700 1 -- 172.21.15.78:0/2373530623 >> 172.21.15.78:0/2373530623 conn(0x7f16640c3a50 msgr2=0x7f16640c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:11.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.372+0000 7f166ae7f700 1 -- 172.21.15.78:0/2373530623 shutdown_connections 2023-10-14T11:59:11.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.372+0000 7f166ae7f700 1 -- 172.21.15.78:0/2373530623 wait complete. 2023-10-14T11:59:11.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.373+0000 7f166ae7f700 1 Processor -- start 2023-10-14T11:59:11.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.373+0000 7f166ae7f700 1 -- start start 2023-10-14T11:59:11.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.374+0000 7f166ae7f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f166406ae10 0x7f16640c6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:11.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.374+0000 7f166ae7f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f16640c4540 0x7f16640ab9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:11.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.374+0000 7f166ae7f700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f166413dbe0 0x7f16640abf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:11.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.374+0000 7f166ae7f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16641e7d60 con 0x7f166406ae10 2023-10-14T11:59:11.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.374+0000 7f166ae7f700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16640a9320 con 0x7f16640c4540 2023-10-14T11:59:11.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.374+0000 7f166ae7f700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1664155fb0 con 0x7f166413dbe0 2023-10-14T11:59:11.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.374+0000 7f1668c1b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f166406ae10 0x7f16640c6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:11.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.374+0000 7f1668c1b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f166406ae10 0x7f16640c6090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46386/0 (socket says 172.21.15.78:46386) 2023-10-14T11:59:11.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.374+0000 7f1668c1b700 1 -- 172.21.15.78:0/3431809890 learned_addr learned my addr 172.21.15.78:0/3431809890 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:11.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.374+0000 7f1668c1b700 1 -- 172.21.15.78:0/3431809890 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f166413dbe0 msgr2=0x7f16640abf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:11.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.374+0000 7f1668c1b700 1 --2- 172.21.15.78:0/3431809890 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f166413dbe0 0x7f16640abf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:11.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.374+0000 7f1668c1b700 1 -- 172.21.15.78:0/3431809890 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f16640c4540 msgr2=0x7f16640ab9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:11.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.374+0000 7f1668c1b700 1 --2- 172.21.15.78:0/3431809890 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f16640c4540 0x7f16640ab9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:11.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.374+0000 7f1668c1b700 1 -- 172.21.15.78:0/3431809890 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1658009550 con 0x7f166406ae10 2023-10-14T11:59:11.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.375+0000 7f1668c1b700 1 --2- 172.21.15.78:0/3431809890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f166406ae10 0x7f16640c6090 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f1664072fc0 tx=0x7f1658004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:11.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.375+0000 7f16617fa700 1 -- 172.21.15.78:0/3431809890 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1658004310 con 0x7f166406ae10 2023-10-14T11:59:11.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.375+0000 7f16617fa700 1 -- 172.21.15.78:0/3431809890 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f165801f070 con 0x7f166406ae10 2023-10-14T11:59:11.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.375+0000 7f166ae7f700 1 -- 172.21.15.78:0/3431809890 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1664156290 con 0x7f166406ae10 2023-10-14T11:59:11.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.375+0000 7f16617fa700 1 -- 172.21.15.78:0/3431809890 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16580036a0 con 0x7f166406ae10 2023-10-14T11:59:11.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.375+0000 7f166ae7f700 1 -- 172.21.15.78:0/3431809890 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1664156850 con 0x7f166406ae10 2023-10-14T11:59:11.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.376+0000 7f166ae7f700 1 -- 172.21.15.78:0/3431809890 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f166406b800 con 0x7f166406ae10 2023-10-14T11:59:11.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.376+0000 7f16617fa700 1 -- 172.21.15.78:0/3431809890 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1658021030 con 0x7f166406ae10 2023-10-14T11:59:11.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.376+0000 7f16617fa700 1 --2- 172.21.15.78:0/3431809890 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f164806af00 0x7f164806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:11.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.377+0000 7f16617fa700 1 -- 172.21.15.78:0/3431809890 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f165808aa80 con 0x7f166406ae10 2023-10-14T11:59:11.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.377+0000 7f1663fff700 1 --2- 172.21.15.78:0/3431809890 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f164806af00 0x7f164806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:11.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.377+0000 7f1663fff700 1 --2- 172.21.15.78:0/3431809890 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f164806af00 0x7f164806d3c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1664152220 tx=0x7f1650008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:11.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.381+0000 7f16617fa700 1 -- 172.21.15.78:0/3431809890 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1658055cc0 con 0x7f166406ae10 2023-10-14T11:59:11.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.649+0000 7f166ae7f700 1 -- 172.21.15.78:0/3431809890 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=testcephfs1", "mds", "allow"]} v 0) v1 -- 0x7f16640c3780 con 0x7f166406ae10 2023-10-14T11:59:11.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.651+0000 7f16617fa700 1 -- 172.21.15.78:0/3431809890 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=testcephfs1", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 195+0+152 (secure 0 0 0) 0x7f16580150f0 con 0x7f166406ae10 2023-10-14T11:59:11.652 INFO:teuthology.orchestra.run.smithi078.stdout:[client.0] 2023-10-14T11:59:11.653 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQAmgiplorThEhAAXHGu+BK6WV8zJzecocBP5Q== 2023-10-14T11:59:11.653 INFO:teuthology.orchestra.run.smithi078.stdout: caps mds = "allow" 2023-10-14T11:59:11.653 INFO:teuthology.orchestra.run.smithi078.stdout: caps mon = "allow r" 2023-10-14T11:59:11.653 INFO:teuthology.orchestra.run.smithi078.stdout: caps osd = "allow rw tag cephfs data=testcephfs1" 2023-10-14T11:59:11.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.653+0000 7f166ae7f700 1 -- 172.21.15.78:0/3431809890 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f164806af00 msgr2=0x7f164806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:11.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.653+0000 7f166ae7f700 1 --2- 172.21.15.78:0/3431809890 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f164806af00 0x7f164806d3c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1664152220 tx=0x7f1650008040 comp rx=0 tx=0).stop 2023-10-14T11:59:11.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.653+0000 7f166ae7f700 1 -- 172.21.15.78:0/3431809890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f166406ae10 msgr2=0x7f16640c6090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:11.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.653+0000 7f166ae7f700 1 --2- 172.21.15.78:0/3431809890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f166406ae10 0x7f16640c6090 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f1664072fc0 tx=0x7f1658004750 comp rx=0 tx=0).stop 2023-10-14T11:59:11.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.653+0000 7f166ae7f700 1 -- 172.21.15.78:0/3431809890 shutdown_connections 2023-10-14T11:59:11.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.653+0000 7f166ae7f700 1 --2- 172.21.15.78:0/3431809890 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f166413dbe0 0x7f16640abf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:11.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.653+0000 7f166ae7f700 1 --2- 172.21.15.78:0/3431809890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f166406ae10 0x7f16640c6090 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:11.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.653+0000 7f166ae7f700 1 --2- 172.21.15.78:0/3431809890 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f164806af00 0x7f164806d3c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:11.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.653+0000 7f166ae7f700 1 --2- 172.21.15.78:0/3431809890 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f16640c4540 0x7f16640ab9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:11.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.653+0000 7f166ae7f700 1 -- 172.21.15.78:0/3431809890 >> 172.21.15.78:0/3431809890 conn(0x7f16640c3a50 msgr2=0x7f16640c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:11.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.653+0000 7f166ae7f700 1 -- 172.21.15.78:0/3431809890 shutdown_connections 2023-10-14T11:59:11.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.653+0000 7f166ae7f700 1 -- 172.21.15.78:0/3431809890 wait complete. 2023-10-14T11:59:11.657 INFO:teuthology.orchestra.run.smithi078.stderr:updated caps for client.0 2023-10-14T11:59:11.667 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T11:59:11.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.784+0000 7f3c8a690700 1 Processor -- start 2023-10-14T11:59:11.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.784+0000 7f3c8a690700 1 -- start start 2023-10-14T11:59:11.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.785+0000 7f3c8a690700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f3c840c44c0 0x7f3c840c7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:11.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.785+0000 7f3c8a690700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3c840c72a0 0x7f3c840c86e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:11.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.785+0000 7f3c8a690700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c840c8c20 0x7f3c840c2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:11.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.785+0000 7f3c8a690700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c84156b20 con 0x7f3c840c72a0 2023-10-14T11:59:11.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.785+0000 7f3c8a690700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c84155f10 con 0x7f3c840c44c0 2023-10-14T11:59:11.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.785+0000 7f3c8a690700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c841561f0 con 0x7f3c840c8c20 2023-10-14T11:59:11.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.785+0000 7f3c837fe700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3c840c72a0 0x7f3c840c86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:11.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.785+0000 7f3c837fe700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3c840c72a0 0x7f3c840c86e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46392/0 (socket says 172.21.15.78:46392) 2023-10-14T11:59:11.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.785+0000 7f3c837fe700 1 -- 172.21.15.78:0/2916170042 learned_addr learned my addr 172.21.15.78:0/2916170042 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:11.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.785+0000 7f3c88c2d700 1 --2- 172.21.15.78:0/2916170042 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c840c8c20 0x7f3c840c2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:11.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.785+0000 7f3c837fe700 1 -- 172.21.15.78:0/2916170042 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c840c8c20 msgr2=0x7f3c840c2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:11.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.786+0000 7f3c837fe700 1 --2- 172.21.15.78:0/2916170042 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c840c8c20 0x7f3c840c2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:11.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.786+0000 7f3c837fe700 1 -- 172.21.15.78:0/2916170042 >> v2:172.21.15.195:3300/0 conn(0x7f3c840c44c0 msgr2=0x7f3c840c7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:11.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.786+0000 7f3c83fff700 1 --2- 172.21.15.78:0/2916170042 >> v2:172.21.15.195:3300/0 conn(0x7f3c840c44c0 0x7f3c840c7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:11.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.786+0000 7f3c837fe700 1 --2- 172.21.15.78:0/2916170042 >> v2:172.21.15.195:3300/0 conn(0x7f3c840c44c0 0x7f3c840c7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:11.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.786+0000 7f3c837fe700 1 -- 172.21.15.78:0/2916170042 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c841564d0 con 0x7f3c840c72a0 2023-10-14T11:59:11.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.786+0000 7f3c837fe700 1 --2- 172.21.15.78:0/2916170042 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c840c72a0 0x7f3c840c86e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3c840c9210 tx=0x7f3c78009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=334ae6d2a7c52842 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:11.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.786+0000 7f3c82ffd700 1 -- 172.21.15.78:0/2916170042 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7800b870 con 0x7f3c840c72a0 2023-10-14T11:59:11.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.786+0000 7f3c82ffd700 1 -- 172.21.15.78:0/2916170042 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c7800b9d0 con 0x7f3c840c72a0 2023-10-14T11:59:11.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.786+0000 7f3c82ffd700 1 -- 172.21.15.78:0/2916170042 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7800bd60 con 0x7f3c840c72a0 2023-10-14T11:59:11.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.786+0000 7f3c8a690700 1 -- 172.21.15.78:0/2916170042 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c840c72a0 msgr2=0x7f3c840c86e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:11.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.786+0000 7f3c8a690700 1 --2- 172.21.15.78:0/2916170042 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c840c72a0 0x7f3c840c86e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3c840c9210 tx=0x7f3c78009bd0 comp rx=0 tx=0).stop 2023-10-14T11:59:11.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.786+0000 7f3c8a690700 1 -- 172.21.15.78:0/2916170042 shutdown_connections 2023-10-14T11:59:11.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.786+0000 7f3c8a690700 1 --2- 172.21.15.78:0/2916170042 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c840c8c20 0x7f3c840c2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:11.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.786+0000 7f3c8a690700 1 --2- 172.21.15.78:0/2916170042 >> v2:172.21.15.195:3300/0 conn(0x7f3c840c44c0 0x7f3c840c7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:11.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.786+0000 7f3c8a690700 1 --2- 172.21.15.78:0/2916170042 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c840c72a0 0x7f3c840c86e0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:11.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.786+0000 7f3c8a690700 1 -- 172.21.15.78:0/2916170042 >> 172.21.15.78:0/2916170042 conn(0x7f3c840737d0 msgr2=0x7f3c84073be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:11.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.786+0000 7f3c8a690700 1 -- 172.21.15.78:0/2916170042 shutdown_connections 2023-10-14T11:59:11.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.787+0000 7f3c8a690700 1 -- 172.21.15.78:0/2916170042 wait complete. 2023-10-14T11:59:11.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.788+0000 7f3c8a690700 1 Processor -- start 2023-10-14T11:59:11.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.788+0000 7f3c8a690700 1 -- start start 2023-10-14T11:59:11.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.788+0000 7f3c8a690700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c840c44c0 0x7f3c840c1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:11.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.788+0000 7f3c8a690700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c840c72a0 0x7f3c8414e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:11.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.788+0000 7f3c8a690700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c840c8c20 0x7f3c840bf230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:11.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.788+0000 7f3c8a690700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c840c10b0 con 0x7f3c840c44c0 2023-10-14T11:59:11.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.788+0000 7f3c8a690700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8414ed40 con 0x7f3c840c8c20 2023-10-14T11:59:11.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.789+0000 7f3c8a690700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8406d1b0 con 0x7f3c840c72a0 2023-10-14T11:59:11.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.789+0000 7f3c83fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c840c44c0 0x7f3c840c1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:11.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.789+0000 7f3c83fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c840c44c0 0x7f3c840c1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46398/0 (socket says 172.21.15.78:46398) 2023-10-14T11:59:11.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.789+0000 7f3c83fff700 1 -- 172.21.15.78:0/2381475457 learned_addr learned my addr 172.21.15.78:0/2381475457 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:11.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.789+0000 7f3c837fe700 1 --2- 172.21.15.78:0/2381475457 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c840c72a0 0x7f3c8414e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:11.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.789+0000 7f3c88c2d700 1 --2- 172.21.15.78:0/2381475457 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c840c8c20 0x7f3c840bf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:11.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.789+0000 7f3c83fff700 1 -- 172.21.15.78:0/2381475457 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c840c72a0 msgr2=0x7f3c8414e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:11.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.789+0000 7f3c83fff700 1 --2- 172.21.15.78:0/2381475457 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c840c72a0 0x7f3c8414e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:11.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.789+0000 7f3c83fff700 1 -- 172.21.15.78:0/2381475457 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c840c8c20 msgr2=0x7f3c840bf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:11.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.789+0000 7f3c83fff700 1 --2- 172.21.15.78:0/2381475457 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c840c8c20 0x7f3c840bf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:11.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.789+0000 7f3c83fff700 1 -- 172.21.15.78:0/2381475457 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c78009550 con 0x7f3c840c44c0 2023-10-14T11:59:11.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.789+0000 7f3c83fff700 1 --2- 172.21.15.78:0/2381475457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c840c44c0 0x7f3c840c1f20 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3c7400c820 tx=0x7f3c7400cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:11.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.789+0000 7f3c80ff9700 1 -- 172.21.15.78:0/2381475457 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c74004220 con 0x7f3c840c44c0 2023-10-14T11:59:11.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.789+0000 7f3c80ff9700 1 -- 172.21.15.78:0/2381475457 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c74004380 con 0x7f3c840c44c0 2023-10-14T11:59:11.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.789+0000 7f3c80ff9700 1 -- 172.21.15.78:0/2381475457 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c740036a0 con 0x7f3c840c44c0 2023-10-14T11:59:11.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.789+0000 7f3c8a690700 1 -- 172.21.15.78:0/2381475457 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c8406c3f0 con 0x7f3c840c44c0 2023-10-14T11:59:11.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.789+0000 7f3c8a690700 1 -- 172.21.15.78:0/2381475457 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c84152650 con 0x7f3c840c44c0 2023-10-14T11:59:11.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.790+0000 7f3c8a690700 1 -- 172.21.15.78:0/2381475457 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c841524b0 con 0x7f3c840c44c0 2023-10-14T11:59:11.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.791+0000 7f3c80ff9700 1 -- 172.21.15.78:0/2381475457 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3c74003800 con 0x7f3c840c44c0 2023-10-14T11:59:11.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.791+0000 7f3c80ff9700 1 --2- 172.21.15.78:0/2381475457 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f3c7006af00 0x7f3c7006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:11.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.791+0000 7f3c80ff9700 1 -- 172.21.15.78:0/2381475457 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f3c74089090 con 0x7f3c840c44c0 2023-10-14T11:59:11.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.791+0000 7f3c837fe700 1 --2- 172.21.15.78:0/2381475457 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f3c7006af00 0x7f3c7006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:11.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.792+0000 7f3c837fe700 1 --2- 172.21.15.78:0/2381475457 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f3c7006af00 0x7f3c7006d3c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3c78000c00 tx=0x7f3c78008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:11.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:11.796+0000 7f3c80ff9700 1 -- 172.21.15.78:0/2381475457 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3c841524b0 con 0x7f3c840c44c0 2023-10-14T11:59:12.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.027+0000 7f3c8a690700 1 -- 172.21.15.78:0/2381475457 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c841524b0 con 0x7f3c840c44c0 2023-10-14T11:59:12.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.027+0000 7f3c80ff9700 1 -- 172.21.15.78:0/2381475457 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4924 (secure 0 0 0) 0x7f3c841524b0 con 0x7f3c840c44c0 2023-10-14T11:59:12.030 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:59:12.030 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4658,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6834/1492486786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1492486786},{"type":"v1","addr":"172.21.15.195:6834","nonce":1492486786}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5016,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/2991384297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2991384297},{"type":"v1","addr":"172.21.15.78:6836","nonce":2991384297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5019,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3246640037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3246640037},{"type":"v1","addr":"172.21.15.78:6837","nonce":3246640037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T11:59:01.562764+0000","modified":"2023-10-14T11:59:04.592935+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5031":{"gid":5031,"name":"d","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.195:6835/2130671508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6833","nonce":2130671508},{"type":"v1","addr":"172.21.15.195:6835","nonce":2130671508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"testcephfs1","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-14T11:59:12.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.029+0000 7f3c8a690700 1 -- 172.21.15.78:0/2381475457 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f3c7006af00 msgr2=0x7f3c7006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:12.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.029+0000 7f3c8a690700 1 --2- 172.21.15.78:0/2381475457 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f3c7006af00 0x7f3c7006d3c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3c78000c00 tx=0x7f3c78008030 comp rx=0 tx=0).stop 2023-10-14T11:59:12.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.029+0000 7f3c8a690700 1 -- 172.21.15.78:0/2381475457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c840c44c0 msgr2=0x7f3c840c1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:12.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.029+0000 7f3c8a690700 1 --2- 172.21.15.78:0/2381475457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c840c44c0 0x7f3c840c1f20 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3c7400c820 tx=0x7f3c7400cb30 comp rx=0 tx=0).stop 2023-10-14T11:59:12.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.029+0000 7f3c8a690700 1 -- 172.21.15.78:0/2381475457 shutdown_connections 2023-10-14T11:59:12.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.029+0000 7f3c8a690700 1 --2- 172.21.15.78:0/2381475457 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f3c7006af00 0x7f3c7006d3c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.029+0000 7f3c8a690700 1 --2- 172.21.15.78:0/2381475457 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c840c8c20 0x7f3c840bf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.029+0000 7f3c8a690700 1 --2- 172.21.15.78:0/2381475457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c840c44c0 0x7f3c840c1f20 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.029+0000 7f3c8a690700 1 --2- 172.21.15.78:0/2381475457 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c840c72a0 0x7f3c8414e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.029+0000 7f3c8a690700 1 -- 172.21.15.78:0/2381475457 >> 172.21.15.78:0/2381475457 conn(0x7f3c840737d0 msgr2=0x7f3c84153360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:12.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.030+0000 7f3c8a690700 1 -- 172.21.15.78:0/2381475457 shutdown_connections 2023-10-14T11:59:12.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.030+0000 7f3c8a690700 1 -- 172.21.15.78:0/2381475457 wait complete. 2023-10-14T11:59:12.034 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 17 2023-10-14T11:59:12.446 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-14T11:59:01.562764+0000', 'modified': '2023-10-14T11:59:04.592935+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5031}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5031': {'gid': 5031, 'name': 'd', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.195:6835/2130671508', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6833', 'nonce': 2130671508}, {'type': 'v1', 'addr': '172.21.15.195:6835', 'nonce': 2130671508}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'testcephfs1', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-14T11:59:12.447 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-14T11:59:12.447 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-14T11:59:12.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.569+0000 7f3e56d26700 1 Processor -- start 2023-10-14T11:59:12.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.569+0000 7f3e56d26700 1 -- start start 2023-10-14T11:59:12.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.570+0000 7f3e56d26700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3e501533d0 0x7f3e500c8680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:12.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.570+0000 7f3e56d26700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f3e500c8bc0 0x7f3e50154960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:12.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.570+0000 7f3e56d26700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e501537b0 con 0x7f3e500a70c0 2023-10-14T11:59:12.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.570+0000 7f3e56d26700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e500c6fb0 con 0x7f3e500c8bc0 2023-10-14T11:59:12.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.570+0000 7f3e56d26700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e500c7f30 con 0x7f3e501533d0 2023-10-14T11:59:12.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.570+0000 7f3e552c3700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f3e500a70c0 0x7f3e50154ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51286/0 (socket says 172.21.15.78:51286) 2023-10-14T11:59:12.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.570+0000 7f3e552c3700 1 -- 172.21.15.78:0/3002555532 learned_addr learned my addr 172.21.15.78:0/3002555532 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:59:12.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.570+0000 7f3e54ac2700 1 --2- 172.21.15.78:0/3002555532 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3e501533d0 0x7f3e500c8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:12.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.570+0000 7f3e4ffff700 1 --2- 172.21.15.78:0/3002555532 >> v2:172.21.15.195:3300/0 conn(0x7f3e500c8bc0 0x7f3e50154960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:12.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.570+0000 7f3e4f7fe700 1 -- 172.21.15.78:0/3002555532 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2584709564 0 0) 0x7f3e501537b0 con 0x7f3e500a70c0 2023-10-14T11:59:12.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.571+0000 7f3e4f7fe700 1 -- 172.21.15.78:0/3002555532 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e400035b0 con 0x7f3e500a70c0 2023-10-14T11:59:12.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.571+0000 7f3e4f7fe700 1 -- 172.21.15.78:0/3002555532 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3919758829 0 0) 0x7f3e400035b0 con 0x7f3e500a70c0 2023-10-14T11:59:12.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.571+0000 7f3e4f7fe700 1 -- 172.21.15.78:0/3002555532 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3e501533d0 msgr2=0x7f3e500c8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:12.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.571+0000 7f3e4f7fe700 1 --2- 172.21.15.78:0/3002555532 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3e501533d0 0x7f3e500c8680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.571+0000 7f3e4f7fe700 1 -- 172.21.15.78:0/3002555532 >> v2:172.21.15.195:3300/0 conn(0x7f3e500c8bc0 msgr2=0x7f3e50154960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:12.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.571+0000 7f3e4f7fe700 1 --2- 172.21.15.78:0/3002555532 >> v2:172.21.15.195:3300/0 conn(0x7f3e500c8bc0 0x7f3e50154960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.571+0000 7f3e4f7fe700 1 -- 172.21.15.78:0/3002555532 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e500c7c50 con 0x7f3e500a70c0 2023-10-14T11:59:12.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.571+0000 7f3e4f7fe700 1 -- 172.21.15.78:0/3002555532 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441799917 0 0) 0x7f3e480019f0 con 0x7f3e500a70c0 2023-10-14T11:59:12.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.571+0000 7f3e4f7fe700 1 -- 172.21.15.78:0/3002555532 >> v1:172.21.15.78:6789/0 conn(0x7f3e500a70c0 legacy=0x7f3e50154ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:12.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.571+0000 7f3e4f7fe700 1 --2- 172.21.15.78:0/3002555532 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3e501533d0 0x7f3e40003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:12.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.571+0000 7f3e4f7fe700 1 --2- 172.21.15.78:0/3002555532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e40003f80 0x7f3e40004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:12.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.571+0000 7f3e4f7fe700 1 --2- 172.21.15.78:0/3002555532 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3e40004d90 0x7f3e40005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:12.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.571+0000 7f3e4f7fe700 1 -- 172.21.15.78:0/3002555532 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e500c6fb0 con 0x7f3e40003f80 2023-10-14T11:59:12.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.571+0000 7f3e4f7fe700 1 -- 172.21.15.78:0/3002555532 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e500c7f30 con 0x7f3e40004d90 2023-10-14T11:59:12.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.571+0000 7f3e4f7fe700 1 -- 172.21.15.78:0/3002555532 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4000ba60 con 0x7f3e501533d0 2023-10-14T11:59:12.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.571+0000 7f3e4ffff700 1 --2- 172.21.15.78:0/3002555532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e40003f80 0x7f3e40004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:12.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.571+0000 7f3e54ac2700 1 --2- 172.21.15.78:0/3002555532 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3e501533d0 0x7f3e40003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:12.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.571+0000 7f3e552c3700 1 --2- 172.21.15.78:0/3002555532 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3e40004d90 0x7f3e40005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:12.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.571+0000 7f3e4ffff700 1 -- 172.21.15.78:0/3002555532 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3e501533d0 msgr2=0x7f3e40003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:12.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.571+0000 7f3e4ffff700 1 --2- 172.21.15.78:0/3002555532 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3e501533d0 0x7f3e40003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.571+0000 7f3e4ffff700 1 -- 172.21.15.78:0/3002555532 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3e40004d90 msgr2=0x7f3e40005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:12.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.571+0000 7f3e4ffff700 1 --2- 172.21.15.78:0/3002555532 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3e40004d90 0x7f3e40005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.571+0000 7f3e4ffff700 1 -- 172.21.15.78:0/3002555532 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e400035b0 con 0x7f3e40003f80 2023-10-14T11:59:12.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.571+0000 7f3e4ffff700 1 --2- 172.21.15.78:0/3002555532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e40003f80 0x7f3e40004850 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c009b90 tx=0x7f3e40005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a96273255763f2a1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:12.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.571+0000 7f3e4f7fe700 1 -- 172.21.15.78:0/3002555532 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c00e6c0 con 0x7f3e40003f80 2023-10-14T11:59:12.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.572+0000 7f3e4f7fe700 1 -- 172.21.15.78:0/3002555532 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e3c00e820 con 0x7f3e40003f80 2023-10-14T11:59:12.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.572+0000 7f3e4f7fe700 1 -- 172.21.15.78:0/3002555532 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c00eb90 con 0x7f3e40003f80 2023-10-14T11:59:12.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.572+0000 7f3e56d26700 1 -- 172.21.15.78:0/3002555532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e40003f80 msgr2=0x7f3e40004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:12.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.572+0000 7f3e56d26700 1 --2- 172.21.15.78:0/3002555532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e40003f80 0x7f3e40004850 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c009b90 tx=0x7f3e40005880 comp rx=0 tx=0).stop 2023-10-14T11:59:12.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.572+0000 7f3e552c3700 1 -- 172.21.15.78:0/3002555532 reap_dead start 2023-10-14T11:59:12.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.572+0000 7f3e56d26700 1 -- 172.21.15.78:0/3002555532 shutdown_connections 2023-10-14T11:59:12.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.572+0000 7f3e56d26700 1 -- 172.21.15.78:0/3002555532 >> 172.21.15.78:0/3002555532 conn(0x7f3e500737d0 msgr2=0x7f3e50073be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:12.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.572+0000 7f3e56d26700 1 -- 172.21.15.78:0/3002555532 shutdown_connections 2023-10-14T11:59:12.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.572+0000 7f3e56d26700 1 -- 172.21.15.78:0/3002555532 wait complete. 2023-10-14T11:59:12.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.574+0000 7f3e56d26700 1 Processor -- start 2023-10-14T11:59:12.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.574+0000 7f3e56d26700 1 -- start start 2023-10-14T11:59:12.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.575+0000 7f3e56d26700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3e40003f80 0x7f3e500c50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:12.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.575+0000 7f3e56d26700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e500c5600 0x7f3e500be6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:12.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.575+0000 7f3e56d26700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3e500bec20 0x7f3e500ab0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:12.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.575+0000 7f3e56d26700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e500c7a70 con 0x7f3e500c5600 2023-10-14T11:59:12.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.575+0000 7f3e56d26700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e500bc250 con 0x7f3e500bec20 2023-10-14T11:59:12.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.575+0000 7f3e56d26700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e500bbc90 con 0x7f3e40003f80 2023-10-14T11:59:12.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.575+0000 7f3e4ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e500c5600 0x7f3e500be6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:12.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.575+0000 7f3e4ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e500c5600 0x7f3e500be6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46420/0 (socket says 172.21.15.78:46420) 2023-10-14T11:59:12.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.575+0000 7f3e4ffff700 1 -- 172.21.15.78:0/2566621043 learned_addr learned my addr 172.21.15.78:0/2566621043 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:12.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.575+0000 7f3e54ac2700 1 --2- 172.21.15.78:0/2566621043 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3e40003f80 0x7f3e500c50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:12.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.575+0000 7f3e552c3700 1 --2- 172.21.15.78:0/2566621043 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3e500bec20 0x7f3e500ab0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:12.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.575+0000 7f3e4ffff700 1 -- 172.21.15.78:0/2566621043 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3e40003f80 msgr2=0x7f3e500c50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:12.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.575+0000 7f3e4ffff700 1 --2- 172.21.15.78:0/2566621043 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3e40003f80 0x7f3e500c50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.575+0000 7f3e4ffff700 1 -- 172.21.15.78:0/2566621043 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3e500bec20 msgr2=0x7f3e500ab0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:12.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.575+0000 7f3e4ffff700 1 --2- 172.21.15.78:0/2566621043 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3e500bec20 0x7f3e500ab0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.575+0000 7f3e4ffff700 1 -- 172.21.15.78:0/2566621043 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e501e84f0 con 0x7f3e500c5600 2023-10-14T11:59:12.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.575+0000 7f3e4ffff700 1 --2- 172.21.15.78:0/2566621043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e500c5600 0x7f3e500be6e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f3e40004790 tx=0x7f3e3c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:12.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.575+0000 7f3e4d7fa700 1 -- 172.21.15.78:0/2566621043 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c021070 con 0x7f3e500c5600 2023-10-14T11:59:12.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.575+0000 7f3e4d7fa700 1 -- 172.21.15.78:0/2566621043 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e3c0045d0 con 0x7f3e500c5600 2023-10-14T11:59:12.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.575+0000 7f3e56d26700 1 -- 172.21.15.78:0/2566621043 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e501e87d0 con 0x7f3e500c5600 2023-10-14T11:59:12.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.575+0000 7f3e4d7fa700 1 -- 172.21.15.78:0/2566621043 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c00e6c0 con 0x7f3e500c5600 2023-10-14T11:59:12.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.576+0000 7f3e56d26700 1 -- 172.21.15.78:0/2566621043 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e500bbee0 con 0x7f3e500c5600 2023-10-14T11:59:12.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.577+0000 7f3e4d7fa700 1 -- 172.21.15.78:0/2566621043 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3e3c025070 con 0x7f3e500c5600 2023-10-14T11:59:12.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.577+0000 7f3e4d7fa700 1 --2- 172.21.15.78:0/2566621043 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f3e3406af00 0x7f3e3406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:12.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.577+0000 7f3e4d7fa700 1 -- 172.21.15.78:0/2566621043 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f3e3c08a6f0 con 0x7f3e500c5600 2023-10-14T11:59:12.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.577+0000 7f3e54ac2700 1 --2- 172.21.15.78:0/2566621043 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f3e3406af00 0x7f3e3406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:12.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.578+0000 7f3e54ac2700 1 --2- 172.21.15.78:0/2566621043 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f3e3406af00 0x7f3e3406d3c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3e44009b80 tx=0x7f3e44009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:12.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.579+0000 7f3e62f10b80 1 Processor -- start 2023-10-14T11:59:12.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.580+0000 7f3e62f10b80 1 -- start start 2023-10-14T11:59:12.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.580+0000 7f3e62f10b80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d62032c6b0 0x55d62036a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:12.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.580+0000 7f3e62f10b80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55d62032ca90 0x55d62036ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:12.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.580+0000 7f3e62f10b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d620326c90 0x55d62034a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:12.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.580+0000 7f3e62f10b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6202a0c20 con 0x55d620326c90 2023-10-14T11:59:12.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.580+0000 7f3e62f10b80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d620285c10 con 0x55d62032c6b0 2023-10-14T11:59:12.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.580+0000 7f3e62f10b80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d620283530 con 0x55d62032ca90 2023-10-14T11:59:12.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.580+0000 7f3e552c3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d620326c90 0x55d62034a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:12.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.580+0000 7f3e552c3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d620326c90 0x55d62034a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46430/0 (socket says 172.21.15.78:46430) 2023-10-14T11:59:12.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.580+0000 7f3e552c3700 1 -- 172.21.15.78:0/2098156617 learned_addr learned my addr 172.21.15.78:0/2098156617 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:12.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.581+0000 7f3e4ffff700 1 --2- 172.21.15.78:0/2098156617 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55d62032ca90 0x55d62036ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:12.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.581+0000 7f3e54ac2700 1 --2- 172.21.15.78:0/2098156617 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d62032c6b0 0x55d62036a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:12.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.581+0000 7f3e552c3700 1 -- 172.21.15.78:0/2098156617 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55d62032ca90 msgr2=0x55d62036ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:12.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.581+0000 7f3e552c3700 1 --2- 172.21.15.78:0/2098156617 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55d62032ca90 0x55d62036ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.581+0000 7f3e552c3700 1 -- 172.21.15.78:0/2098156617 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d62032c6b0 msgr2=0x55d62036a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:12.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.581+0000 7f3e552c3700 1 --2- 172.21.15.78:0/2098156617 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d62032c6b0 0x55d62036a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.581+0000 7f3e552c3700 1 -- 172.21.15.78:0/2098156617 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d62023c3a0 con 0x55d620326c90 2023-10-14T11:59:12.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.581+0000 7f3e4ffff700 1 --2- 172.21.15.78:0/2098156617 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55d62032ca90 0x55d62036ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:59:12.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.581+0000 7f3e552c3700 1 --2- 172.21.15.78:0/2098156617 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d620326c90 0x55d62034a8a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3e50156dc0 tx=0x7f3e48007330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76e87eb0d80af461 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:12.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.581+0000 7f3e3a7fc700 1 -- 172.21.15.78:0/2098156617 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e48007940 con 0x55d620326c90 2023-10-14T11:59:12.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.581+0000 7f3e3a7fc700 1 -- 172.21.15.78:0/2098156617 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e48007aa0 con 0x55d620326c90 2023-10-14T11:59:12.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.581+0000 7f3e3a7fc700 1 -- 172.21.15.78:0/2098156617 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e48014600 con 0x55d620326c90 2023-10-14T11:59:12.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.582+0000 7f3e62f10b80 1 -- 172.21.15.78:0/2098156617 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d620326c90 msgr2=0x55d62034a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:12.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.582+0000 7f3e62f10b80 1 --2- 172.21.15.78:0/2098156617 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d620326c90 0x55d62034a8a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3e50156dc0 tx=0x7f3e48007330 comp rx=0 tx=0).stop 2023-10-14T11:59:12.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.582+0000 7f3e62f10b80 1 -- 172.21.15.78:0/2098156617 shutdown_connections 2023-10-14T11:59:12.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.582+0000 7f3e62f10b80 1 --2- 172.21.15.78:0/2098156617 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d620326c90 0x55d62034a8a0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.582+0000 7f3e62f10b80 1 --2- 172.21.15.78:0/2098156617 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d62032c6b0 0x55d62036a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.582+0000 7f3e62f10b80 1 --2- 172.21.15.78:0/2098156617 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55d62032ca90 0x55d62036ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.582+0000 7f3e62f10b80 1 -- 172.21.15.78:0/2098156617 >> 172.21.15.78:0/2098156617 conn(0x55d620259d80 msgr2=0x55d62032a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:12.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.582+0000 7f3e62f10b80 1 -- 172.21.15.78:0/2098156617 shutdown_connections 2023-10-14T11:59:12.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.582+0000 7f3e62f10b80 1 -- 172.21.15.78:0/2098156617 wait complete. 2023-10-14T11:59:12.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.582+0000 7f3e62f10b80 1 -- start start 2023-10-14T11:59:12.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.582+0000 7f3e62f10b80 1 Processor -- start 2023-10-14T11:59:12.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.583+0000 7f3e62f10b80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d620326c90 0x55d62034c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:12.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.583+0000 7f3e62f10b80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55d62032c6b0 0x55d62036a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:12.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.583+0000 7f3e62f10b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d62032ca90 0x55d6203442b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:12.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.583+0000 7f3e62f10b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d62027ee20 con 0x55d62032ca90 2023-10-14T11:59:12.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.583+0000 7f3e62f10b80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d62021b6c0 con 0x55d620326c90 2023-10-14T11:59:12.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.583+0000 7f3e62f10b80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d62021b360 con 0x55d62032c6b0 2023-10-14T11:59:12.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.583+0000 7f3e62f10b80 10 client.0 _ll_register_callbacks cb 0x55d6200787b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T11:59:12.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.583+0000 7f3e552c3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d62032ca90 0x55d6203442b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:12.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.583+0000 7f3e552c3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d62032ca90 0x55d6203442b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46436/0 (socket says 172.21.15.78:46436) 2023-10-14T11:59:12.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.583+0000 7f3e552c3700 1 -- 172.21.15.78:0/2775123086 learned_addr learned my addr 172.21.15.78:0/2775123086 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:12.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.583+0000 7f3e54ac2700 1 --2- 172.21.15.78:0/2775123086 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d620326c90 0x55d62034c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:12.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.583+0000 7f3e4ffff700 1 --2- 172.21.15.78:0/2775123086 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55d62032c6b0 0x55d62036a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:12.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.584+0000 7f3e552c3700 1 -- 172.21.15.78:0/2775123086 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55d62032c6b0 msgr2=0x55d62036a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:12.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.584+0000 7f3e552c3700 1 --2- 172.21.15.78:0/2775123086 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55d62032c6b0 0x55d62036a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.584+0000 7f3e552c3700 1 -- 172.21.15.78:0/2775123086 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d620326c90 msgr2=0x55d62034c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:12.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.584+0000 7f3e552c3700 1 --2- 172.21.15.78:0/2775123086 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d620326c90 0x55d62034c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.584+0000 7f3e552c3700 1 -- 172.21.15.78:0/2775123086 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e4800f040 con 0x55d62032ca90 2023-10-14T11:59:12.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.584+0000 7f3e552c3700 1 --2- 172.21.15.78:0/2775123086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d62032ca90 0x55d6203442b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f3e48000f80 tx=0x7f3e48001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15c61116a5ffcbfd server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:12.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.584+0000 7f3e2f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-14T11:59:12.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.584+0000 7f3e2f7fe700 1 -- 172.21.15.78:0/2775123086 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e48007940 con 0x55d62032ca90 2023-10-14T11:59:12.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.584+0000 7f3e2f7fe700 1 -- 172.21.15.78:0/2775123086 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f3e48014d40 con 0x55d62032ca90 2023-10-14T11:59:12.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.584+0000 7f3e62f10b80 1 -- 172.21.15.78:0/2775123086 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d62031f5b0 con 0x55d62032ca90 2023-10-14T11:59:12.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.584+0000 7f3e2f7fe700 1 -- 172.21.15.78:0/2775123086 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e4801f410 con 0x55d62032ca90 2023-10-14T11:59:12.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.584+0000 7f3e62f10b80 10 client.5241 fetch_fsmap learned FSMap version 17 2023-10-14T11:59:12.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.584+0000 7f3e62f10b80 1 -- 172.21.15.78:0/2775123086 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55d6200a9d90 con 0x55d62032ca90 2023-10-14T11:59:12.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.584+0000 7f3e2f7fe700 1 -- 172.21.15.78:0/2775123086 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55d62031f5b0 con 0x55d62032ca90 2023-10-14T11:59:12.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.584+0000 7f3e2f7fe700 1 -- 172.21.15.78:0/2775123086 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 17) v1 ==== 3299+0+0 (secure 0 0 0) 0x7f3e4801f570 con 0x55d62032ca90 2023-10-14T11:59:12.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.584+0000 7f3e62f10b80 10 client.5241 fetch_fsmap finished waiting for FSMap version 17 2023-10-14T11:59:12.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.584+0000 7f3e62f10b80 10 client.5241 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-14T11:59:12.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.584+0000 7f3e62f10b80 20 client.5241 populate_metadata read hostname 'smithi078' 2023-10-14T11:59:12.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.584+0000 7f3e62f10b80 1 --2- 172.21.15.78:0/2775123086 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x55d6203448c0 0x55d620378220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:12.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.584+0000 7f3e62f10b80 4 client.5241 mds_command: new command op to 5031 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-14T11:59:12.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.584+0000 7f3e62f10b80 1 -- 172.21.15.78:0/2775123086 --> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d620277e00 con 0x55d6203448c0 2023-10-14T11:59:12.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.584+0000 7f3e54ac2700 1 --2- 172.21.15.78:0/2775123086 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x55d6203448c0 0x55d620378220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:12.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.585+0000 7f3e2f7fe700 10 client.5241 ms_handle_connect on v2:172.21.15.195:6833/2130671508 2023-10-14T11:59:12.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.585+0000 7f3e54ac2700 1 --2- 172.21.15.78:0/2775123086 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x55d6203448c0 0x55d620378220 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22125213b94c5a62 server_cookie=3a44c675096ffbe3 in_seq=0 out_seq=0 2023-10-14T11:59:12.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.586+0000 7f3e2f7fe700 1 -- 172.21.15.78:0/2775123086 <== mds.0 v2:172.21.15.195:6833/2130671508 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55d620277e00 con 0x55d6203448c0 2023-10-14T11:59:12.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.586+0000 7f3e2f7fe700 10 client.5241 handle_command_reply: tid=0 2023-10-14T11:59:12.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.586+0000 7f3e62f10b80 1 client.5241 shutdown 2023-10-14T11:59:12.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.586+0000 7f3e62f10b80 1 -- 172.21.15.78:0/2775123086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d62032ca90 msgr2=0x55d6203442b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:12.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.586+0000 7f3e62f10b80 1 --2- 172.21.15.78:0/2775123086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d62032ca90 0x55d6203442b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f3e48000f80 tx=0x7f3e48001d80 comp rx=0 tx=0).stop 2023-10-14T11:59:12.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.586+0000 7f3e62f10b80 1 -- 172.21.15.78:0/2775123086 shutdown_connections 2023-10-14T11:59:12.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.586+0000 7f3e62f10b80 1 --2- 172.21.15.78:0/2775123086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d62032ca90 0x55d6203442b0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.586+0000 7f3e62f10b80 1 --2- 172.21.15.78:0/2775123086 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x55d6203448c0 0x55d620378220 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.586+0000 7f3e62f10b80 1 --2- 172.21.15.78:0/2775123086 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d620326c90 0x55d62034c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.586+0000 7f3e2f7fe700 0 client.5241 ms_handle_reset on v2:172.21.15.195:6833/2130671508 2023-10-14T11:59:12.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.586+0000 7f3e62f10b80 1 --2- 172.21.15.78:0/2775123086 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55d62032c6b0 0x55d62036a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.586+0000 7f3e62f10b80 1 -- 172.21.15.78:0/2775123086 >> 172.21.15.78:0/2775123086 conn(0x55d620259d80 msgr2=0x55d6203217d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:12.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.586+0000 7f3e62f10b80 1 -- 172.21.15.78:0/2775123086 shutdown_connections 2023-10-14T11:59:12.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.586+0000 7f3e62f10b80 1 -- 172.21.15.78:0/2775123086 wait complete. 2023-10-14T11:59:12.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.586+0000 7f3e62f10b80 20 client.5241 trim_cache size 0 max 16384 2023-10-14T11:59:12.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.603+0000 7f3e62f10b80 1 Processor -- start 2023-10-14T11:59:12.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.603+0000 7f3e62f10b80 1 -- start start 2023-10-14T11:59:12.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.604+0000 7f3e62f10b80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55d620326c90 0x55d6203750b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:12.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.604+0000 7f3e62f10b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d62032c6b0 0x55d6203755f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:12.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.604+0000 7f3e62f10b80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d62032ca90 0x55d620378220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:12.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.604+0000 7f3e62f10b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6200ff8d0 con 0x55d62032c6b0 2023-10-14T11:59:12.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.604+0000 7f3e62f10b80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d62036a7c0 con 0x55d62032ca90 2023-10-14T11:59:12.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.604+0000 7f3e62f10b80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d6203023c0 con 0x55d620326c90 2023-10-14T11:59:12.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.604+0000 7f3e4ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d62032c6b0 0x55d6203755f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:12.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.604+0000 7f3e4ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d62032c6b0 0x55d6203755f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46444/0 (socket says 172.21.15.78:46444) 2023-10-14T11:59:12.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.604+0000 7f3e4ffff700 1 -- 172.21.15.78:0/512534873 learned_addr learned my addr 172.21.15.78:0/512534873 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:12.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.604+0000 7f3e4ffff700 1 -- 172.21.15.78:0/512534873 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55d620326c90 msgr2=0x55d6203750b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:12.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.604+0000 7f3e54ac2700 1 --2- 172.21.15.78:0/512534873 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55d620326c90 0x55d6203750b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:12.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.604+0000 7f3e4ffff700 1 --2- 172.21.15.78:0/512534873 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55d620326c90 0x55d6203750b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.604+0000 7f3e4ffff700 1 -- 172.21.15.78:0/512534873 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d62032ca90 msgr2=0x55d620378220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:12.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.604+0000 7f3e4ffff700 1 --2- 172.21.15.78:0/512534873 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d62032ca90 0x55d620378220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.604+0000 7f3e4ffff700 1 -- 172.21.15.78:0/512534873 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d620092c10 con 0x55d62032c6b0 2023-10-14T11:59:12.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.605+0000 7f3e4ffff700 1 --2- 172.21.15.78:0/512534873 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d62032c6b0 0x55d6203755f0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c0559d0 tx=0x7f3e3c059350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2850334cb170ee9 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:12.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.605+0000 7f3e3affd700 1 -- 172.21.15.78:0/512534873 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c0617d0 con 0x55d62032c6b0 2023-10-14T11:59:12.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.605+0000 7f3e3affd700 1 -- 172.21.15.78:0/512534873 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e3c057870 con 0x55d62032c6b0 2023-10-14T11:59:12.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.605+0000 7f3e3affd700 1 -- 172.21.15.78:0/512534873 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c009850 con 0x55d62032c6b0 2023-10-14T11:59:12.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.605+0000 7f3e62f10b80 1 -- 172.21.15.78:0/512534873 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d62032c6b0 msgr2=0x55d6203755f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:12.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.605+0000 7f3e62f10b80 1 --2- 172.21.15.78:0/512534873 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d62032c6b0 0x55d6203755f0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c0559d0 tx=0x7f3e3c059350 comp rx=0 tx=0).stop 2023-10-14T11:59:12.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.605+0000 7f3e62f10b80 1 -- 172.21.15.78:0/512534873 shutdown_connections 2023-10-14T11:59:12.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.605+0000 7f3e62f10b80 1 --2- 172.21.15.78:0/512534873 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d62032ca90 0x55d620378220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.605+0000 7f3e62f10b80 1 --2- 172.21.15.78:0/512534873 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55d620326c90 0x55d6203750b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.605+0000 7f3e62f10b80 1 --2- 172.21.15.78:0/512534873 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d62032c6b0 0x55d6203755f0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.605+0000 7f3e62f10b80 1 -- 172.21.15.78:0/512534873 >> 172.21.15.78:0/512534873 conn(0x55d620259d80 msgr2=0x55d62038a980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:12.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.605+0000 7f3e62f10b80 1 -- 172.21.15.78:0/512534873 shutdown_connections 2023-10-14T11:59:12.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.605+0000 7f3e62f10b80 1 -- 172.21.15.78:0/512534873 wait complete. 2023-10-14T11:59:12.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.606+0000 7f3e62f10b80 1 -- start start 2023-10-14T11:59:12.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.606+0000 7f3e62f10b80 1 Processor -- start 2023-10-14T11:59:12.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.607+0000 7f3e62f10b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d620326c90 0x55d620378620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:12.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.607+0000 7f3e62f10b80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d62032c6b0 0x55d6203d38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:12.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.607+0000 7f3e62f10b80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55d62032ca90 0x55d6203d3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:12.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.607+0000 7f3e62f10b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d620378b60 con 0x55d620326c90 2023-10-14T11:59:12.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.607+0000 7f3e62f10b80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6203d4360 con 0x55d62032c6b0 2023-10-14T11:59:12.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.607+0000 7f3e62f10b80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d6203d44d0 con 0x55d62032ca90 2023-10-14T11:59:12.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.607+0000 7f3e62f10b80 10 client.0 _ll_register_callbacks cb 0x7f3e480019e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T11:59:12.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.607+0000 7f3e54ac2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d620326c90 0x55d620378620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:12.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.607+0000 7f3e54ac2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d620326c90 0x55d620378620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46446/0 (socket says 172.21.15.78:46446) 2023-10-14T11:59:12.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.607+0000 7f3e54ac2700 1 -- 172.21.15.78:0/644400847 learned_addr learned my addr 172.21.15.78:0/644400847 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:12.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.607+0000 7f3e552c3700 1 --2- 172.21.15.78:0/644400847 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55d62032ca90 0x55d6203d3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:12.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.607+0000 7f3e4ffff700 1 --2- 172.21.15.78:0/644400847 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d62032c6b0 0x55d6203d38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:12.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.607+0000 7f3e54ac2700 1 -- 172.21.15.78:0/644400847 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55d62032ca90 msgr2=0x55d6203d3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:12.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.607+0000 7f3e54ac2700 1 --2- 172.21.15.78:0/644400847 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55d62032ca90 0x55d6203d3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.607+0000 7f3e54ac2700 1 -- 172.21.15.78:0/644400847 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d62032c6b0 msgr2=0x55d6203d38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:12.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.607+0000 7f3e54ac2700 1 --2- 172.21.15.78:0/644400847 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d62032c6b0 0x55d6203d38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.607+0000 7f3e54ac2700 1 -- 172.21.15.78:0/644400847 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e500bbee0 con 0x55d620326c90 2023-10-14T11:59:12.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.607+0000 7f3e552c3700 1 --2- 172.21.15.78:0/644400847 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55d62032ca90 0x55d6203d3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:59:12.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.607+0000 7f3e54ac2700 1 --2- 172.21.15.78:0/644400847 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d620326c90 0x55d620378620 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f3e44003900 tx=0x7f3e44004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa266a0bd761458b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:12.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.607+0000 7f3e2f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-14T11:59:12.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.607+0000 7f3e2f7fe700 1 -- 172.21.15.78:0/644400847 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4402a070 con 0x55d620326c90 2023-10-14T11:59:12.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.607+0000 7f3e2f7fe700 1 -- 172.21.15.78:0/644400847 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f3e4402b460 con 0x55d620326c90 2023-10-14T11:59:12.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.607+0000 7f3e62f10b80 1 -- 172.21.15.78:0/644400847 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d62034d090 con 0x55d620326c90 2023-10-14T11:59:12.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.607+0000 7f3e2f7fe700 1 -- 172.21.15.78:0/644400847 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e4402b7a0 con 0x55d620326c90 2023-10-14T11:59:12.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.607+0000 7f3e62f10b80 10 client.5247 fetch_fsmap learned FSMap version 17 2023-10-14T11:59:12.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.607+0000 7f3e62f10b80 1 -- 172.21.15.78:0/644400847 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55d6200d0200 con 0x55d620326c90 2023-10-14T11:59:12.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.608+0000 7f3e2f7fe700 1 -- 172.21.15.78:0/644400847 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55d62034d090 con 0x55d620326c90 2023-10-14T11:59:12.633 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.608+0000 7f3e2f7fe700 1 -- 172.21.15.78:0/644400847 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 17) v1 ==== 3299+0+0 (secure 0 0 0) 0x7f3e4402b900 con 0x55d620326c90 2023-10-14T11:59:12.633 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.608+0000 7f3e62f10b80 10 client.5247 fetch_fsmap finished waiting for FSMap version 17 2023-10-14T11:59:12.633 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.608+0000 7f3e62f10b80 10 client.5247 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-14T11:59:12.633 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.608+0000 7f3e62f10b80 20 client.5247 populate_metadata read hostname 'smithi078' 2023-10-14T11:59:12.633 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.608+0000 7f3e62f10b80 1 --2- 172.21.15.78:0/644400847 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x55d6203cdcb0 0x55d620320f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:12.633 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.608+0000 7f3e62f10b80 4 client.5247 mds_command: new command op to 5031 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-14T11:59:12.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.608+0000 7f3e62f10b80 1 -- 172.21.15.78:0/644400847 --> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f3e48007aa0 con 0x55d6203cdcb0 2023-10-14T11:59:12.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.608+0000 7f3e4ffff700 1 --2- 172.21.15.78:0/644400847 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x55d6203cdcb0 0x55d620320f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:12.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.608+0000 7f3e2f7fe700 10 client.5247 ms_handle_connect on v2:172.21.15.195:6833/2130671508 2023-10-14T11:59:12.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.608+0000 7f3e4ffff700 1 --2- 172.21.15.78:0/644400847 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x55d6203cdcb0 0x55d620320f40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2db1b3e5246ce15d server_cookie=f46c4d3112d23004 in_seq=0 out_seq=0 2023-10-14T11:59:12.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.608+0000 7f3e2f7fe700 1 -- 172.21.15.78:0/644400847 <== mds.0 v2:172.21.15.195:6833/2130671508 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+336 (crc 0 0 0) 0x7f3e48007aa0 con 0x55d6203cdcb0 2023-10-14T11:59:12.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.608+0000 7f3e2f7fe700 10 client.5247 handle_command_reply: tid=0 2023-10-14T11:59:12.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.608+0000 7f3e62f10b80 1 client.5247 shutdown 2023-10-14T11:59:12.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.609+0000 7f3e62f10b80 1 -- 172.21.15.78:0/644400847 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d620326c90 msgr2=0x55d620378620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:12.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.609+0000 7f3e62f10b80 1 --2- 172.21.15.78:0/644400847 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d620326c90 0x55d620378620 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f3e44003900 tx=0x7f3e44004750 comp rx=0 tx=0).stop 2023-10-14T11:59:12.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.609+0000 7f3e62f10b80 1 -- 172.21.15.78:0/644400847 shutdown_connections 2023-10-14T11:59:12.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.609+0000 7f3e62f10b80 1 --2- 172.21.15.78:0/644400847 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55d62032ca90 0x55d6203d3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.609+0000 7f3e62f10b80 1 --2- 172.21.15.78:0/644400847 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d620326c90 0x55d620378620 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.609+0000 7f3e62f10b80 1 --2- 172.21.15.78:0/644400847 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x55d6203cdcb0 0x55d620320f40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.609+0000 7f3e62f10b80 1 --2- 172.21.15.78:0/644400847 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d62032c6b0 0x55d6203d38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.609+0000 7f3e62f10b80 1 -- 172.21.15.78:0/644400847 >> 172.21.15.78:0/644400847 conn(0x55d620259d80 msgr2=0x55d62036c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:12.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.609+0000 7f3e2f7fe700 0 client.5247 ms_handle_reset on v2:172.21.15.195:6833/2130671508 2023-10-14T11:59:12.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.609+0000 7f3e62f10b80 1 -- 172.21.15.78:0/644400847 shutdown_connections 2023-10-14T11:59:12.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.609+0000 7f3e62f10b80 1 -- 172.21.15.78:0/644400847 wait complete. 2023-10-14T11:59:12.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.609+0000 7f3e62f10b80 20 client.5247 trim_cache size 0 max 16384 2023-10-14T11:59:12.638 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-14T11:59:12.638 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "4c5dc134-3092-406a-a1e1-13e6d9e926e8", 2023-10-14T11:59:12.638 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2023-10-14T11:59:12.638 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 5031, 2023-10-14T11:59:12.638 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2023-10-14T11:59:12.639 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2023-10-14T11:59:12.639 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "testcephfs1", 2023-10-14T11:59:12.639 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 10.424620946999999, 2023-10-14T11:59:12.639 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 17, 2023-10-14T11:59:12.639 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 32, 2023-10-14T11:59:12.639 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 32, 2023-10-14T11:59:12.640 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 18.315446190999999 2023-10-14T11:59:12.640 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-14T11:59:12.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.609+0000 7f3e56d26700 1 -- 172.21.15.78:0/2566621043 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f3e3406af00 msgr2=0x7f3e3406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:12.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.609+0000 7f3e56d26700 1 --2- 172.21.15.78:0/2566621043 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f3e3406af00 0x7f3e3406d3c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3e44009b80 tx=0x7f3e44009210 comp rx=0 tx=0).stop 2023-10-14T11:59:12.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.609+0000 7f3e56d26700 1 -- 172.21.15.78:0/2566621043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e500c5600 msgr2=0x7f3e500be6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:12.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.609+0000 7f3e56d26700 1 --2- 172.21.15.78:0/2566621043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e500c5600 0x7f3e500be6e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f3e40004790 tx=0x7f3e3c0047c0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.609+0000 7f3e56d26700 1 -- 172.21.15.78:0/2566621043 shutdown_connections 2023-10-14T11:59:12.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.609+0000 7f3e56d26700 1 --2- 172.21.15.78:0/2566621043 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f3e3406af00 0x7f3e3406d3c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.609+0000 7f3e56d26700 1 --2- 172.21.15.78:0/2566621043 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3e500bec20 0x7f3e500ab0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.609+0000 7f3e56d26700 1 --2- 172.21.15.78:0/2566621043 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3e40003f80 0x7f3e500c50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.609+0000 7f3e56d26700 1 --2- 172.21.15.78:0/2566621043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e500c5600 0x7f3e500be6e0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.609+0000 7f3e56d26700 1 -- 172.21.15.78:0/2566621043 >> 172.21.15.78:0/2566621043 conn(0x7f3e500737d0 msgr2=0x7f3e5014ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:12.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.609+0000 7f3e56d26700 1 -- 172.21.15.78:0/2566621043 shutdown_connections 2023-10-14T11:59:12.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.609+0000 7f3e56d26700 1 -- 172.21.15.78:0/2566621043 wait complete. 2023-10-14T11:59:12.644 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T11:59:12.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.763+0000 7f2cf7323700 1 Processor -- start 2023-10-14T11:59:12.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.764+0000 7f2cf7323700 1 -- start start 2023-10-14T11:59:12.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.764+0000 7f2cf7323700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f2cf014b6b0 0x7f2cf0072580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:12.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.764+0000 7f2cf7323700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2cf0154bb0 0x7f2cf0155030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:12.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.764+0000 7f2cf7323700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cf00c25e0 con 0x7f2cf006db50 2023-10-14T11:59:12.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.764+0000 7f2cf7323700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cf00a87b0 con 0x7f2cf014b6b0 2023-10-14T11:59:12.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.764+0000 7f2cf7323700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf014ba90 con 0x7f2cf0154bb0 2023-10-14T11:59:12.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.765+0000 7f2cf58c0700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2cf0154bb0 0x7f2cf0155030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:12.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.765+0000 7f2cf50bf700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f2cf014b6b0 0x7f2cf0072580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:12.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.765+0000 7f2cf58c0700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2cf0154bb0 0x7f2cf0155030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.78:46126/0 (socket says 172.21.15.78:46126) 2023-10-14T11:59:12.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.765+0000 7f2cf58c0700 1 -- 172.21.15.78:0/428791022 learned_addr learned my addr 172.21.15.78:0/428791022 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:12.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.765+0000 7f2cf58c0700 1 -- 172.21.15.78:0/428791022 >> v2:172.21.15.195:3300/0 conn(0x7f2cf014b6b0 msgr2=0x7f2cf0072580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:12.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.765+0000 7f2cf58c0700 1 --2- 172.21.15.78:0/428791022 >> v2:172.21.15.195:3300/0 conn(0x7f2cf014b6b0 0x7f2cf0072580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.765+0000 7f2cf58c0700 1 -- 172.21.15.78:0/428791022 >> v1:172.21.15.195:6789/0 conn(0x7f2cf006db50 legacy=0x7f2cf00c2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:12.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.765+0000 7f2cf58c0700 1 -- 172.21.15.78:0/428791022 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf0155660 con 0x7f2cf0154bb0 2023-10-14T11:59:12.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.765+0000 7f2cf58c0700 1 --2- 172.21.15.78:0/428791022 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2cf0154bb0 0x7f2cf0155030 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8009e50 tx=0x7f2ce800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=904e675a558f8f3a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:12.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.765+0000 7f2ce7fff700 1 -- 172.21.15.78:0/428791022 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce800e070 con 0x7f2cf0154bb0 2023-10-14T11:59:12.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.765+0000 7f2ce7fff700 1 -- 172.21.15.78:0/428791022 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ce800b910 con 0x7f2cf0154bb0 2023-10-14T11:59:12.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.766+0000 7f2ce7fff700 1 -- 172.21.15.78:0/428791022 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce800bcc0 con 0x7f2cf0154bb0 2023-10-14T11:59:12.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.766+0000 7f2cf7323700 1 -- 172.21.15.78:0/428791022 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2cf0154bb0 msgr2=0x7f2cf0155030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:12.772 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.766+0000 7f2cf7323700 1 --2- 172.21.15.78:0/428791022 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2cf0154bb0 0x7f2cf0155030 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8009e50 tx=0x7f2ce800b330 comp rx=0 tx=0).stop 2023-10-14T11:59:12.772 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.766+0000 7f2cf7323700 1 -- 172.21.15.78:0/428791022 shutdown_connections 2023-10-14T11:59:12.772 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.766+0000 7f2cf7323700 1 --2- 172.21.15.78:0/428791022 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2cf0154bb0 0x7f2cf0155030 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.772 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.766+0000 7f2cf7323700 1 --2- 172.21.15.78:0/428791022 >> v2:172.21.15.195:3300/0 conn(0x7f2cf014b6b0 0x7f2cf0072580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.772 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.766+0000 7f2cf7323700 1 -- 172.21.15.78:0/428791022 >> 172.21.15.78:0/428791022 conn(0x7f2cf00c3ac0 msgr2=0x7f2cf00c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:12.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.766+0000 7f2cf7323700 1 -- 172.21.15.78:0/428791022 shutdown_connections 2023-10-14T11:59:12.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.766+0000 7f2cf7323700 1 -- 172.21.15.78:0/428791022 wait complete. 2023-10-14T11:59:12.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.767+0000 7f2cf7323700 1 Processor -- start 2023-10-14T11:59:12.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.767+0000 7f2cf7323700 1 -- start start 2023-10-14T11:59:12.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.768+0000 7f2cf7323700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cf006db50 0x7f2cf01df2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:12.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.768+0000 7f2cf7323700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2cf014b6b0 0x7f2cf0149780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:12.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.768+0000 7f2cf7323700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2cf0154bb0 0x7f2cf0149cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:12.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.768+0000 7f2cf7323700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf01edce0 con 0x7f2cf006db50 2023-10-14T11:59:12.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.768+0000 7f2cf7323700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf01ed060 con 0x7f2cf0154bb0 2023-10-14T11:59:12.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.768+0000 7f2cf7323700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf01ed340 con 0x7f2cf014b6b0 2023-10-14T11:59:12.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.768+0000 7f2cf50bf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cf006db50 0x7f2cf01df2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:12.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.768+0000 7f2cf50bf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cf006db50 0x7f2cf01df2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46458/0 (socket says 172.21.15.78:46458) 2023-10-14T11:59:12.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.768+0000 7f2cf50bf700 1 -- 172.21.15.78:0/3242224420 learned_addr learned my addr 172.21.15.78:0/3242224420 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:12.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.768+0000 7f2cf48be700 1 --2- 172.21.15.78:0/3242224420 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2cf014b6b0 0x7f2cf0149780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:12.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.768+0000 7f2cf58c0700 1 --2- 172.21.15.78:0/3242224420 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2cf0154bb0 0x7f2cf0149cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:12.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.768+0000 7f2cf50bf700 1 -- 172.21.15.78:0/3242224420 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2cf014b6b0 msgr2=0x7f2cf0149780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:12.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.768+0000 7f2cf50bf700 1 --2- 172.21.15.78:0/3242224420 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2cf014b6b0 0x7f2cf0149780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.768+0000 7f2cf50bf700 1 -- 172.21.15.78:0/3242224420 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2cf0154bb0 msgr2=0x7f2cf0149cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:12.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.768+0000 7f2cf50bf700 1 --2- 172.21.15.78:0/3242224420 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2cf0154bb0 0x7f2cf0149cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:12.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.768+0000 7f2cf50bf700 1 -- 172.21.15.78:0/3242224420 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce8009b30 con 0x7f2cf006db50 2023-10-14T11:59:12.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.768+0000 7f2cf50bf700 1 --2- 172.21.15.78:0/3242224420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cf006db50 0x7f2cf01df2a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f2ce000e9b0 tx=0x7f2ce000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:12.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.769+0000 7f2ce5ffb700 1 -- 172.21.15.78:0/3242224420 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce0009710 con 0x7f2cf006db50 2023-10-14T11:59:12.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.769+0000 7f2ce5ffb700 1 -- 172.21.15.78:0/3242224420 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ce000c850 con 0x7f2cf006db50 2023-10-14T11:59:12.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.769+0000 7f2cf7323700 1 -- 172.21.15.78:0/3242224420 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cf01ed620 con 0x7f2cf006db50 2023-10-14T11:59:12.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.769+0000 7f2ce5ffb700 1 -- 172.21.15.78:0/3242224420 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce000cbc0 con 0x7f2cf006db50 2023-10-14T11:59:12.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.769+0000 7f2cf7323700 1 -- 172.21.15.78:0/3242224420 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cf01df7e0 con 0x7f2cf006db50 2023-10-14T11:59:12.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.770+0000 7f2cf7323700 1 -- 172.21.15.78:0/3242224420 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cf0074300 con 0x7f2cf006db50 2023-10-14T11:59:12.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.770+0000 7f2ce5ffb700 1 -- 172.21.15.78:0/3242224420 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2ce000cd20 con 0x7f2cf006db50 2023-10-14T11:59:12.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.771+0000 7f2ce5ffb700 1 --2- 172.21.15.78:0/3242224420 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f2cd406af00 0x7f2cd406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:12.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.771+0000 7f2cf48be700 1 --2- 172.21.15.78:0/3242224420 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f2cd406af00 0x7f2cd406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:12.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.771+0000 7f2ce5ffb700 1 -- 172.21.15.78:0/3242224420 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f2ce00896d0 con 0x7f2cf006db50 2023-10-14T11:59:12.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.771+0000 7f2cf48be700 1 --2- 172.21.15.78:0/3242224420 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f2cd406af00 0x7f2cd406d3c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2cf014aee0 tx=0x7f2cec009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:12.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:12.775+0000 7f2ce5ffb700 1 -- 172.21.15.78:0/3242224420 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2ce00547f0 con 0x7f2cf006db50 2023-10-14T11:59:13.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.006+0000 7f2cf7323700 1 -- 172.21.15.78:0/3242224420 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2cf0074b80 con 0x7f2cf006db50 2023-10-14T11:59:13.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.006+0000 7f2ce5ffb700 1 -- 172.21.15.78:0/3242224420 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4924 (secure 0 0 0) 0x7f2ce0057e10 con 0x7f2cf006db50 2023-10-14T11:59:13.009 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:59:13.009 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4658,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6834/1492486786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1492486786},{"type":"v1","addr":"172.21.15.195:6834","nonce":1492486786}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5016,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/2991384297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2991384297},{"type":"v1","addr":"172.21.15.78:6836","nonce":2991384297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5019,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3246640037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3246640037},{"type":"v1","addr":"172.21.15.78:6837","nonce":3246640037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T11:59:01.562764+0000","modified":"2023-10-14T11:59:04.592935+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5031":{"gid":5031,"name":"d","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.195:6835/2130671508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6833","nonce":2130671508},{"type":"v1","addr":"172.21.15.195:6835","nonce":2130671508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"testcephfs1","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-14T11:59:13.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.008+0000 7f2cf7323700 1 -- 172.21.15.78:0/3242224420 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f2cd406af00 msgr2=0x7f2cd406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:13.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.008+0000 7f2cf7323700 1 --2- 172.21.15.78:0/3242224420 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f2cd406af00 0x7f2cd406d3c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2cf014aee0 tx=0x7f2cec009040 comp rx=0 tx=0).stop 2023-10-14T11:59:13.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.008+0000 7f2cf7323700 1 -- 172.21.15.78:0/3242224420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cf006db50 msgr2=0x7f2cf01df2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:13.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.008+0000 7f2cf7323700 1 --2- 172.21.15.78:0/3242224420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cf006db50 0x7f2cf01df2a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f2ce000e9b0 tx=0x7f2ce000ed70 comp rx=0 tx=0).stop 2023-10-14T11:59:13.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.008+0000 7f2cf7323700 1 -- 172.21.15.78:0/3242224420 shutdown_connections 2023-10-14T11:59:13.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.008+0000 7f2cf7323700 1 --2- 172.21.15.78:0/3242224420 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f2cd406af00 0x7f2cd406d3c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:13.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.008+0000 7f2cf7323700 1 --2- 172.21.15.78:0/3242224420 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2cf0154bb0 0x7f2cf0149cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:13.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.008+0000 7f2cf7323700 1 --2- 172.21.15.78:0/3242224420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cf006db50 0x7f2cf01df2a0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:13.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.008+0000 7f2cf7323700 1 --2- 172.21.15.78:0/3242224420 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2cf014b6b0 0x7f2cf0149780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:13.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.008+0000 7f2cf7323700 1 -- 172.21.15.78:0/3242224420 >> 172.21.15.78:0/3242224420 conn(0x7f2cf00c3ac0 msgr2=0x7f2cf006a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:13.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.008+0000 7f2cf7323700 1 -- 172.21.15.78:0/3242224420 shutdown_connections 2023-10-14T11:59:13.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.008+0000 7f2cf7323700 1 -- 172.21.15.78:0/3242224420 wait complete. 2023-10-14T11:59:13.013 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 17 2023-10-14T11:59:13.023 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 17, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-14T11:59:01.562764+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 17, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "testcephfs1", "in": [ 0 ], "info": { "gid_5031": { "addr": "172.21.15.195:6835/2130671508", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6833", "nonce": 2130671508, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2130671508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5031, "incarnation": 13, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2023-10-14T11:59:04.592935+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5031 } } } ], "standbys": [ { "addr": "172.21.15.195:6834/1492486786", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6832", "nonce": 1492486786, "type": "v2" }, { "addr": "172.21.15.195:6834", "nonce": 1492486786, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4658, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6836/2991384297", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6834", "nonce": 2991384297, "type": "v2" }, { "addr": "172.21.15.78:6836", "nonce": 2991384297, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5016, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6837/3246640037", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6835", "nonce": 3246640037, "type": "v2" }, { "addr": "172.21.15.78:6837", "nonce": 3246640037, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5019, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-14T11:59:13.023 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2023-10-14T11:59:13.024 INFO:tasks.cephfs.test_failover:current = [{'gid': 5019, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6837/3246640037', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 3246640037}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 3246640037}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 4658, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.195:6834/1492486786', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6832', 'nonce': 1492486786}, {'type': 'v1', 'addr': '172.21.15.195:6834', 'nonce': 1492486786}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 5016, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6836/2991384297', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 2991384297}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 2991384297}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 5031, 'name': 'd', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.195:6835/2130671508', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6833', 'nonce': 2130671508}, {'type': 'v1', 'addr': '172.21.15.195:6835', 'nonce': 2130671508}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2023-10-14T11:59:13.024 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.b mds_join_fs cephfs 2023-10-14T11:59:13.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.145+0000 7fe426234700 1 Processor -- start 2023-10-14T11:59:13.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.145+0000 7fe426234700 1 -- start start 2023-10-14T11:59:13.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.146+0000 7fe426234700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe420070070 0x7fe420152630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:13.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.146+0000 7fe426234700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe42006f9c0 0x7fe420152b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:13.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.146+0000 7fe426234700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe420156e40 con 0x7fe420152250 2023-10-14T11:59:13.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.146+0000 7fe426234700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4200c3360 con 0x7fe420070070 2023-10-14T11:59:13.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.146+0000 7fe426234700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe420156080 con 0x7fe42006f9c0 2023-10-14T11:59:13.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.147+0000 7fe41ffff700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe42006f9c0 0x7fe420152b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:13.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.147+0000 7fe41effd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe420070070 0x7fe420152630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:13.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.147+0000 7fe41ffff700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe42006f9c0 0x7fe420152b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.78:46140/0 (socket says 172.21.15.78:46140) 2023-10-14T11:59:13.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.147+0000 7fe41ffff700 1 -- 172.21.15.78:0/3133225294 learned_addr learned my addr 172.21.15.78:0/3133225294 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:13.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.147+0000 7fe41ffff700 1 -- 172.21.15.78:0/3133225294 >> v2:172.21.15.195:3300/0 conn(0x7fe420070070 msgr2=0x7fe420152630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:13.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.147+0000 7fe41ffff700 1 --2- 172.21.15.78:0/3133225294 >> v2:172.21.15.195:3300/0 conn(0x7fe420070070 0x7fe420152630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:13.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.147+0000 7fe41ffff700 1 -- 172.21.15.78:0/3133225294 >> v1:172.21.15.195:6789/0 conn(0x7fe420152250 legacy=0x7fe4200c3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:13.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.147+0000 7fe41ffff700 1 -- 172.21.15.78:0/3133225294 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe420156360 con 0x7fe42006f9c0 2023-10-14T11:59:13.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.147+0000 7fe41ffff700 1 --2- 172.21.15.78:0/3133225294 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe42006f9c0 0x7fe420152b70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe418009e50 tx=0x7fe41800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56fcb102f3a113b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:13.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.147+0000 7fe41e7fc700 1 -- 172.21.15.78:0/3133225294 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41800e070 con 0x7fe42006f9c0 2023-10-14T11:59:13.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.147+0000 7fe41e7fc700 1 -- 172.21.15.78:0/3133225294 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe41800b910 con 0x7fe42006f9c0 2023-10-14T11:59:13.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.147+0000 7fe41e7fc700 1 -- 172.21.15.78:0/3133225294 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41800bcc0 con 0x7fe42006f9c0 2023-10-14T11:59:13.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.148+0000 7fe426234700 1 -- 172.21.15.78:0/3133225294 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe42006f9c0 msgr2=0x7fe420152b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:13.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.148+0000 7fe426234700 1 --2- 172.21.15.78:0/3133225294 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe42006f9c0 0x7fe420152b70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe418009e50 tx=0x7fe41800b330 comp rx=0 tx=0).stop 2023-10-14T11:59:13.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.148+0000 7fe426234700 1 -- 172.21.15.78:0/3133225294 shutdown_connections 2023-10-14T11:59:13.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.148+0000 7fe426234700 1 --2- 172.21.15.78:0/3133225294 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe42006f9c0 0x7fe420152b70 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:13.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.148+0000 7fe426234700 1 --2- 172.21.15.78:0/3133225294 >> v2:172.21.15.195:3300/0 conn(0x7fe420070070 0x7fe420152630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:13.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.148+0000 7fe426234700 1 -- 172.21.15.78:0/3133225294 >> 172.21.15.78:0/3133225294 conn(0x7fe4200733e0 msgr2=0x7fe4200737c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:13.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.148+0000 7fe426234700 1 -- 172.21.15.78:0/3133225294 shutdown_connections 2023-10-14T11:59:13.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.148+0000 7fe426234700 1 -- 172.21.15.78:0/3133225294 wait complete. 2023-10-14T11:59:13.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.149+0000 7fe426234700 1 Processor -- start 2023-10-14T11:59:13.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.149+0000 7fe426234700 1 -- start start 2023-10-14T11:59:13.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.150+0000 7fe426234700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe42006f9c0 0x7fe420150610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:13.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.150+0000 7fe426234700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe420152250 0x7fe42014cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:13.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.150+0000 7fe426234700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe42006d4b0 0x7fe42014d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:13.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.150+0000 7fe426234700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4200bbfe0 con 0x7fe420152250 2023-10-14T11:59:13.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.150+0000 7fe426234700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe42014d7e0 con 0x7fe42006d4b0 2023-10-14T11:59:13.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.150+0000 7fe426234700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe420155060 con 0x7fe42006f9c0 2023-10-14T11:59:13.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.150+0000 7fe41effd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe420152250 0x7fe42014cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:13.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.150+0000 7fe41effd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe420152250 0x7fe42014cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46474/0 (socket says 172.21.15.78:46474) 2023-10-14T11:59:13.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.150+0000 7fe41effd700 1 -- 172.21.15.78:0/3110453714 learned_addr learned my addr 172.21.15.78:0/3110453714 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:13.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.150+0000 7fe41f7fe700 1 --2- 172.21.15.78:0/3110453714 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe42006f9c0 0x7fe420150610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:13.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.150+0000 7fe41ffff700 1 --2- 172.21.15.78:0/3110453714 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe42006d4b0 0x7fe42014d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:13.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.150+0000 7fe41effd700 1 -- 172.21.15.78:0/3110453714 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe42006f9c0 msgr2=0x7fe420150610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:13.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.150+0000 7fe41effd700 1 --2- 172.21.15.78:0/3110453714 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe42006f9c0 0x7fe420150610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:13.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.150+0000 7fe41effd700 1 -- 172.21.15.78:0/3110453714 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe42006d4b0 msgr2=0x7fe42014d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:13.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.150+0000 7fe41effd700 1 --2- 172.21.15.78:0/3110453714 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe42006d4b0 0x7fe42014d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:13.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.150+0000 7fe41effd700 1 -- 172.21.15.78:0/3110453714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe418009b30 con 0x7fe420152250 2023-10-14T11:59:13.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.150+0000 7fe41effd700 1 --2- 172.21.15.78:0/3110453714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe420152250 0x7fe42014cd60 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe40800e9b0 tx=0x7fe40800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:13.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.151+0000 7fe417fff700 1 -- 172.21.15.78:0/3110453714 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe408009710 con 0x7fe420152250 2023-10-14T11:59:13.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.151+0000 7fe417fff700 1 -- 172.21.15.78:0/3110453714 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe40800ee50 con 0x7fe420152250 2023-10-14T11:59:13.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.151+0000 7fe426234700 1 -- 172.21.15.78:0/3110453714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4201ea030 con 0x7fe420152250 2023-10-14T11:59:13.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.151+0000 7fe417fff700 1 -- 172.21.15.78:0/3110453714 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40800ca50 con 0x7fe420152250 2023-10-14T11:59:13.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.151+0000 7fe426234700 1 -- 172.21.15.78:0/3110453714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4201ea5f0 con 0x7fe420152250 2023-10-14T11:59:13.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.152+0000 7fe426234700 1 -- 172.21.15.78:0/3110453714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4200ac040 con 0x7fe420152250 2023-10-14T11:59:13.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.152+0000 7fe417fff700 1 -- 172.21.15.78:0/3110453714 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe40800cbb0 con 0x7fe420152250 2023-10-14T11:59:13.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.152+0000 7fe417fff700 1 --2- 172.21.15.78:0/3110453714 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe40006aed0 0x7fe40006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:13.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.153+0000 7fe41f7fe700 1 --2- 172.21.15.78:0/3110453714 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe40006aed0 0x7fe40006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:13.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.153+0000 7fe417fff700 1 -- 172.21.15.78:0/3110453714 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7fe408089590 con 0x7fe420152250 2023-10-14T11:59:13.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.153+0000 7fe41f7fe700 1 --2- 172.21.15.78:0/3110453714 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe40006aed0 0x7fe40006d390 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe4200bc890 tx=0x7fe410009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:13.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.157+0000 7fe417fff700 1 -- 172.21.15.78:0/3110453714 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe408054630 con 0x7fe420152250 2023-10-14T11:59:13.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.350+0000 7fe426234700 1 -- 172.21.15.78:0/3110453714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7fe4200c8750 con 0x7fe420152250 2023-10-14T11:59:13.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.353+0000 7fe417fff700 1 -- 172.21.15.78:0/3110453714 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v12)=0 v12) v1 ==== 116+0+0 (secure 0 0 0) 0x7fe40800ce60 con 0x7fe420152250 2023-10-14T11:59:13.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.355+0000 7fe426234700 1 -- 172.21.15.78:0/3110453714 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe40006aed0 msgr2=0x7fe40006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:13.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.355+0000 7fe426234700 1 --2- 172.21.15.78:0/3110453714 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe40006aed0 0x7fe40006d390 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe4200bc890 tx=0x7fe410009040 comp rx=0 tx=0).stop 2023-10-14T11:59:13.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.355+0000 7fe426234700 1 -- 172.21.15.78:0/3110453714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe420152250 msgr2=0x7fe42014cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:13.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.355+0000 7fe426234700 1 --2- 172.21.15.78:0/3110453714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe420152250 0x7fe42014cd60 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe40800e9b0 tx=0x7fe40800ecc0 comp rx=0 tx=0).stop 2023-10-14T11:59:13.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.355+0000 7fe426234700 1 -- 172.21.15.78:0/3110453714 shutdown_connections 2023-10-14T11:59:13.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.355+0000 7fe426234700 1 --2- 172.21.15.78:0/3110453714 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe40006aed0 0x7fe40006d390 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:13.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.355+0000 7fe426234700 1 --2- 172.21.15.78:0/3110453714 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe42006d4b0 0x7fe42014d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:13.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.355+0000 7fe426234700 1 --2- 172.21.15.78:0/3110453714 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe42006f9c0 0x7fe420150610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:13.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.355+0000 7fe426234700 1 --2- 172.21.15.78:0/3110453714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe420152250 0x7fe42014cd60 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:13.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.355+0000 7fe426234700 1 -- 172.21.15.78:0/3110453714 >> 172.21.15.78:0/3110453714 conn(0x7fe4200733e0 msgr2=0x7fe420154300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:13.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.355+0000 7fe426234700 1 -- 172.21.15.78:0/3110453714 shutdown_connections 2023-10-14T11:59:13.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.355+0000 7fe426234700 1 -- 172.21.15.78:0/3110453714 wait complete. 2023-10-14T11:59:13.369 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T11:59:13.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.487+0000 7f8cf3a67700 1 Processor -- start 2023-10-14T11:59:13.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.487+0000 7f8cf3a67700 1 -- start start 2023-10-14T11:59:13.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.488+0000 7f8cf3a67700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f8cec14ed30 0x7f8cec0c23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:13.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.488+0000 7f8cf3a67700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8cec152d00 0x7f8cec1531b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:13.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.488+0000 7f8cf3a67700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8cec1536f0 0x7f8cec154bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:13.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.488+0000 7f8cf3a67700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cec0c3590 con 0x7f8cec152d00 2023-10-14T11:59:13.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.488+0000 7f8cf3a67700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cec0724b0 con 0x7f8cec14ed30 2023-10-14T11:59:13.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.488+0000 7f8cf3a67700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec0c2980 con 0x7f8cec1536f0 2023-10-14T11:59:13.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.488+0000 7f8cf1002700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8cec152d00 0x7f8cec1531b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:13.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.489+0000 7f8cf1002700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8cec152d00 0x7f8cec1531b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46482/0 (socket says 172.21.15.78:46482) 2023-10-14T11:59:13.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.489+0000 7f8cf1002700 1 -- 172.21.15.78:0/3944526851 learned_addr learned my addr 172.21.15.78:0/3944526851 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:13.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.489+0000 7f8cf2004700 1 --2- 172.21.15.78:0/3944526851 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8cec1536f0 0x7f8cec154bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:13.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.489+0000 7f8cf1803700 1 --2- 172.21.15.78:0/3944526851 >> v2:172.21.15.195:3300/0 conn(0x7f8cec14ed30 0x7f8cec0c23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:13.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.489+0000 7f8cf1002700 1 -- 172.21.15.78:0/3944526851 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8cec1536f0 msgr2=0x7f8cec154bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:13.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.489+0000 7f8cf1002700 1 --2- 172.21.15.78:0/3944526851 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8cec1536f0 0x7f8cec154bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:13.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.489+0000 7f8cf1002700 1 -- 172.21.15.78:0/3944526851 >> v2:172.21.15.195:3300/0 conn(0x7f8cec14ed30 msgr2=0x7f8cec0c23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:13.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.489+0000 7f8cf1002700 1 --2- 172.21.15.78:0/3944526851 >> v2:172.21.15.195:3300/0 conn(0x7f8cec14ed30 0x7f8cec0c23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:13.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.489+0000 7f8cf1002700 1 -- 172.21.15.78:0/3944526851 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cec0c2c60 con 0x7f8cec152d00 2023-10-14T11:59:13.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.489+0000 7f8cf1803700 1 --2- 172.21.15.78:0/3944526851 >> v2:172.21.15.195:3300/0 conn(0x7f8cec14ed30 0x7f8cec0c23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:59:13.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.489+0000 7f8cf1002700 1 --2- 172.21.15.78:0/3944526851 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cec152d00 0x7f8cec1531b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8cec0c46d0 tx=0x7f8ce8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=735519d1e2d1ab40 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:13.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.489+0000 7f8cf0801700 1 -- 172.21.15.78:0/3944526851 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce800b870 con 0x7f8cec152d00 2023-10-14T11:59:13.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.489+0000 7f8cf0801700 1 -- 172.21.15.78:0/3944526851 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ce800b9d0 con 0x7f8cec152d00 2023-10-14T11:59:13.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.489+0000 7f8cf0801700 1 -- 172.21.15.78:0/3944526851 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce800bd60 con 0x7f8cec152d00 2023-10-14T11:59:13.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.490+0000 7f8cf3a67700 1 -- 172.21.15.78:0/3944526851 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cec152d00 msgr2=0x7f8cec1531b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:13.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.490+0000 7f8cf3a67700 1 --2- 172.21.15.78:0/3944526851 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cec152d00 0x7f8cec1531b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8cec0c46d0 tx=0x7f8ce8009bd0 comp rx=0 tx=0).stop 2023-10-14T11:59:13.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.490+0000 7f8cf3a67700 1 -- 172.21.15.78:0/3944526851 shutdown_connections 2023-10-14T11:59:13.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.490+0000 7f8cf3a67700 1 --2- 172.21.15.78:0/3944526851 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8cec1536f0 0x7f8cec154bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:13.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.490+0000 7f8cf3a67700 1 --2- 172.21.15.78:0/3944526851 >> v2:172.21.15.195:3300/0 conn(0x7f8cec14ed30 0x7f8cec0c23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:13.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.490+0000 7f8cf3a67700 1 --2- 172.21.15.78:0/3944526851 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cec152d00 0x7f8cec1531b0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:13.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.490+0000 7f8cf3a67700 1 -- 172.21.15.78:0/3944526851 >> 172.21.15.78:0/3944526851 conn(0x7f8cec0c3a90 msgr2=0x7f8cec0c3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:13.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.490+0000 7f8cf3a67700 1 -- 172.21.15.78:0/3944526851 shutdown_connections 2023-10-14T11:59:13.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.490+0000 7f8cf3a67700 1 -- 172.21.15.78:0/3944526851 wait complete. 2023-10-14T11:59:13.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.491+0000 7f8cf3a67700 1 Processor -- start 2023-10-14T11:59:13.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.491+0000 7f8cf3a67700 1 -- start start 2023-10-14T11:59:13.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.492+0000 7f8cf3a67700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cec14ed30 0x7f8cec0c6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:13.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.492+0000 7f8cf3a67700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8cec152d00 0x7f8cec0c56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:13.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.492+0000 7f8cf3a67700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8cec1536f0 0x7f8cec1e9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:13.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.492+0000 7f8cf3a67700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec1ed1b0 con 0x7f8cec14ed30 2023-10-14T11:59:13.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.492+0000 7f8cf3a67700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec1ed490 con 0x7f8cec1536f0 2023-10-14T11:59:13.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.492+0000 7f8cf3a67700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec1ed770 con 0x7f8cec152d00 2023-10-14T11:59:13.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.492+0000 7f8cf1803700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cec14ed30 0x7f8cec0c6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:13.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.492+0000 7f8cf1803700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cec14ed30 0x7f8cec0c6770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46486/0 (socket says 172.21.15.78:46486) 2023-10-14T11:59:13.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.492+0000 7f8cf1803700 1 -- 172.21.15.78:0/2291643514 learned_addr learned my addr 172.21.15.78:0/2291643514 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:13.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.492+0000 7f8cf1002700 1 --2- 172.21.15.78:0/2291643514 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8cec152d00 0x7f8cec0c56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:13.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.492+0000 7f8cf2004700 1 --2- 172.21.15.78:0/2291643514 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8cec1536f0 0x7f8cec1e9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:13.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.492+0000 7f8cf1803700 1 -- 172.21.15.78:0/2291643514 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8cec152d00 msgr2=0x7f8cec0c56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:13.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.492+0000 7f8cf1803700 1 --2- 172.21.15.78:0/2291643514 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8cec152d00 0x7f8cec0c56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:13.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.492+0000 7f8cf1803700 1 -- 172.21.15.78:0/2291643514 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8cec1536f0 msgr2=0x7f8cec1e9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:13.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.492+0000 7f8cf1803700 1 --2- 172.21.15.78:0/2291643514 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8cec1536f0 0x7f8cec1e9cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:13.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.492+0000 7f8cf1803700 1 -- 172.21.15.78:0/2291643514 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce8009550 con 0x7f8cec14ed30 2023-10-14T11:59:13.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.493+0000 7f8cf1803700 1 --2- 172.21.15.78:0/2291643514 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cec14ed30 0x7f8cec0c6770 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f8cdc00c750 tx=0x7f8cdc00cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:13.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.493+0000 7f8ce27fc700 1 -- 172.21.15.78:0/2291643514 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cdc011800 con 0x7f8cec14ed30 2023-10-14T11:59:13.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.493+0000 7f8ce27fc700 1 -- 172.21.15.78:0/2291643514 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cdc011960 con 0x7f8cec14ed30 2023-10-14T11:59:13.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.493+0000 7f8cf3a67700 1 -- 172.21.15.78:0/2291643514 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cec1eda50 con 0x7f8cec14ed30 2023-10-14T11:59:13.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.493+0000 7f8ce27fc700 1 -- 172.21.15.78:0/2291643514 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cdc011cd0 con 0x7f8cec14ed30 2023-10-14T11:59:13.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.493+0000 7f8cf3a67700 1 -- 172.21.15.78:0/2291643514 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cec0c6cb0 con 0x7f8cec14ed30 2023-10-14T11:59:13.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.494+0000 7f8cf3a67700 1 -- 172.21.15.78:0/2291643514 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cec074b80 con 0x7f8cec14ed30 2023-10-14T11:59:13.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.494+0000 7f8ce27fc700 1 -- 172.21.15.78:0/2291643514 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8cdc021350 con 0x7f8cec14ed30 2023-10-14T11:59:13.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.495+0000 7f8ce27fc700 1 --2- 172.21.15.78:0/2291643514 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8cd006af00 0x7f8cd006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:13.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.495+0000 7f8cf1002700 1 --2- 172.21.15.78:0/2291643514 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8cd006af00 0x7f8cd006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:13.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.495+0000 7f8ce27fc700 1 -- 172.21.15.78:0/2291643514 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f8cdc0894b0 con 0x7f8cec14ed30 2023-10-14T11:59:13.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.495+0000 7f8cf1002700 1 --2- 172.21.15.78:0/2291643514 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8cd006af00 0x7f8cd006d3c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8000c00 tx=0x7f8ce8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:13.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.499+0000 7f8ce27fc700 1 -- 172.21.15.78:0/2291643514 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8cdc0545d0 con 0x7f8cec14ed30 2023-10-14T11:59:13.731 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.729+0000 7f8cf3a67700 1 -- 172.21.15.78:0/2291643514 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8cec0c4980 con 0x7f8cec14ed30 2023-10-14T11:59:13.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.730+0000 7f8ce27fc700 1 -- 172.21.15.78:0/2291643514 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4924 (secure 0 0 0) 0x7f8cdc057bf0 con 0x7f8cec14ed30 2023-10-14T11:59:13.732 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:59:13.732 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4658,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6834/1492486786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1492486786},{"type":"v1","addr":"172.21.15.195:6834","nonce":1492486786}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5016,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/2991384297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2991384297},{"type":"v1","addr":"172.21.15.78:6836","nonce":2991384297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5019,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3246640037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3246640037},{"type":"v1","addr":"172.21.15.78:6837","nonce":3246640037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T11:59:01.562764+0000","modified":"2023-10-14T11:59:04.592935+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5031":{"gid":5031,"name":"d","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.195:6835/2130671508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6833","nonce":2130671508},{"type":"v1","addr":"172.21.15.195:6835","nonce":2130671508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"testcephfs1","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-14T11:59:13.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.732+0000 7f8cf3a67700 1 -- 172.21.15.78:0/2291643514 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8cd006af00 msgr2=0x7f8cd006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:13.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.732+0000 7f8cf3a67700 1 --2- 172.21.15.78:0/2291643514 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8cd006af00 0x7f8cd006d3c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8000c00 tx=0x7f8ce8008030 comp rx=0 tx=0).stop 2023-10-14T11:59:13.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.732+0000 7f8cf3a67700 1 -- 172.21.15.78:0/2291643514 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cec14ed30 msgr2=0x7f8cec0c6770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:13.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.732+0000 7f8cf3a67700 1 --2- 172.21.15.78:0/2291643514 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cec14ed30 0x7f8cec0c6770 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f8cdc00c750 tx=0x7f8cdc00cb10 comp rx=0 tx=0).stop 2023-10-14T11:59:13.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.732+0000 7f8cf3a67700 1 -- 172.21.15.78:0/2291643514 shutdown_connections 2023-10-14T11:59:13.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.732+0000 7f8cf3a67700 1 --2- 172.21.15.78:0/2291643514 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8cd006af00 0x7f8cd006d3c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:13.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.732+0000 7f8cf3a67700 1 --2- 172.21.15.78:0/2291643514 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8cec1536f0 0x7f8cec1e9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:13.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.732+0000 7f8cf3a67700 1 --2- 172.21.15.78:0/2291643514 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cec14ed30 0x7f8cec0c6770 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:13.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.732+0000 7f8cf3a67700 1 --2- 172.21.15.78:0/2291643514 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8cec152d00 0x7f8cec0c56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:13.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.732+0000 7f8cf3a67700 1 -- 172.21.15.78:0/2291643514 >> 172.21.15.78:0/2291643514 conn(0x7f8cec0c3a90 msgr2=0x7f8cec14e000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:13.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.732+0000 7f8cf3a67700 1 -- 172.21.15.78:0/2291643514 shutdown_connections 2023-10-14T11:59:13.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:13.732+0000 7f8cf3a67700 1 -- 172.21.15.78:0/2291643514 wait complete. 2023-10-14T11:59:13.737 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 17 2023-10-14T11:59:14.149 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-14T11:59:01.562764+0000', 'modified': '2023-10-14T11:59:04.592935+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5031}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5031': {'gid': 5031, 'name': 'd', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.195:6835/2130671508', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6833', 'nonce': 2130671508}, {'type': 'v1', 'addr': '172.21.15.195:6835', 'nonce': 2130671508}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'testcephfs1', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-14T11:59:14.149 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-14T11:59:14.150 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-14T11:59:14.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.273+0000 7f75b17c4700 1 Processor -- start 2023-10-14T11:59:14.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.273+0000 7f75b17c4700 1 -- start start 2023-10-14T11:59:14.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.274+0000 7f75b17c4700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f75ac0c4540 0x7f75ac0c02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:14.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.274+0000 7f75b17c4700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f75ac06b500 0x7f75ac0c0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:14.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.274+0000 7f75b17c4700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75ac06ebe0 con 0x7f75ac13dbe0 2023-10-14T11:59:14.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.274+0000 7f75b17c4700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75ac06d330 con 0x7f75ac0c4540 2023-10-14T11:59:14.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.274+0000 7f75b17c4700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac0c0d60 con 0x7f75ac06b500 2023-10-14T11:59:14.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.275+0000 7f75aa7fc700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f75ac0c4540 0x7f75ac0c02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:14.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.275+0000 7f75ab7fe700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f75ac06b500 0x7f75ac0c0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:14.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.275+0000 7f75ab7fe700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f75ac06b500 0x7f75ac0c0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.78:46168/0 (socket says 172.21.15.78:46168) 2023-10-14T11:59:14.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.275+0000 7f75ab7fe700 1 -- 172.21.15.78:0/3639858967 learned_addr learned my addr 172.21.15.78:0/3639858967 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:14.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.275+0000 7f75ab7fe700 1 -- 172.21.15.78:0/3639858967 >> v2:172.21.15.195:3300/0 conn(0x7f75ac0c4540 msgr2=0x7f75ac0c02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:14.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.275+0000 7f75ab7fe700 1 --2- 172.21.15.78:0/3639858967 >> v2:172.21.15.195:3300/0 conn(0x7f75ac0c4540 0x7f75ac0c02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:14.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.275+0000 7f75ab7fe700 1 -- 172.21.15.78:0/3639858967 >> v1:172.21.15.195:6789/0 conn(0x7f75ac13dbe0 legacy=0x7f75ac06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:14.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.275+0000 7f75ab7fe700 1 -- 172.21.15.78:0/3639858967 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75ac1558c0 con 0x7f75ac06b500 2023-10-14T11:59:14.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.275+0000 7f75ab7fe700 1 --2- 172.21.15.78:0/3639858967 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f75ac06b500 0x7f75ac0c0820 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f75a4009e50 tx=0x7f75a400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da3b447b69302cf server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:14.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.275+0000 7f75a9ffb700 1 -- 172.21.15.78:0/3639858967 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a400e070 con 0x7f75ac06b500 2023-10-14T11:59:14.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.275+0000 7f75a9ffb700 1 -- 172.21.15.78:0/3639858967 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75a400b910 con 0x7f75ac06b500 2023-10-14T11:59:14.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.275+0000 7f75a9ffb700 1 -- 172.21.15.78:0/3639858967 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a400bcc0 con 0x7f75ac06b500 2023-10-14T11:59:14.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.276+0000 7f75b17c4700 1 -- 172.21.15.78:0/3639858967 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f75ac06b500 msgr2=0x7f75ac0c0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:14.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.276+0000 7f75b17c4700 1 --2- 172.21.15.78:0/3639858967 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f75ac06b500 0x7f75ac0c0820 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f75a4009e50 tx=0x7f75a400b330 comp rx=0 tx=0).stop 2023-10-14T11:59:14.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.276+0000 7f75b17c4700 1 -- 172.21.15.78:0/3639858967 shutdown_connections 2023-10-14T11:59:14.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.276+0000 7f75b17c4700 1 --2- 172.21.15.78:0/3639858967 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f75ac06b500 0x7f75ac0c0820 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:14.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.276+0000 7f75b17c4700 1 --2- 172.21.15.78:0/3639858967 >> v2:172.21.15.195:3300/0 conn(0x7f75ac0c4540 0x7f75ac0c02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:14.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.276+0000 7f75b17c4700 1 -- 172.21.15.78:0/3639858967 >> 172.21.15.78:0/3639858967 conn(0x7f75ac0c3a50 msgr2=0x7f75ac0c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:14.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.276+0000 7f75b17c4700 1 -- 172.21.15.78:0/3639858967 shutdown_connections 2023-10-14T11:59:14.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.276+0000 7f75b17c4700 1 -- 172.21.15.78:0/3639858967 wait complete. 2023-10-14T11:59:14.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.277+0000 7f75b17c4700 1 Processor -- start 2023-10-14T11:59:14.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.277+0000 7f75b17c4700 1 -- start start 2023-10-14T11:59:14.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.278+0000 7f75b17c4700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75ac06b500 0x7f75ac158900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:14.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.278+0000 7f75b17c4700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f75ac0c4540 0x7f75ac1564b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:14.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.278+0000 7f75b17c4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75ac13dbe0 0x7f75ac13b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:14.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.278+0000 7f75b17c4700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac1384c0 con 0x7f75ac13dbe0 2023-10-14T11:59:14.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.278+0000 7f75b17c4700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac158e40 con 0x7f75ac06b500 2023-10-14T11:59:14.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.278+0000 7f75b17c4700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac0a8a50 con 0x7f75ac0c4540 2023-10-14T11:59:14.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.278+0000 7f75ab7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75ac13dbe0 0x7f75ac13b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:14.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.278+0000 7f75ab7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75ac13dbe0 0x7f75ac13b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46504/0 (socket says 172.21.15.78:46504) 2023-10-14T11:59:14.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.278+0000 7f75ab7fe700 1 -- 172.21.15.78:0/2657940194 learned_addr learned my addr 172.21.15.78:0/2657940194 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:14.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.278+0000 7f75aaffd700 1 --2- 172.21.15.78:0/2657940194 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75ac06b500 0x7f75ac158900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:14.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.278+0000 7f75aa7fc700 1 --2- 172.21.15.78:0/2657940194 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f75ac0c4540 0x7f75ac1564b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:14.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.278+0000 7f75ab7fe700 1 -- 172.21.15.78:0/2657940194 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f75ac0c4540 msgr2=0x7f75ac1564b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:14.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.278+0000 7f75ab7fe700 1 --2- 172.21.15.78:0/2657940194 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f75ac0c4540 0x7f75ac1564b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:14.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.278+0000 7f75ab7fe700 1 -- 172.21.15.78:0/2657940194 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75ac06b500 msgr2=0x7f75ac158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:14.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.278+0000 7f75ab7fe700 1 --2- 172.21.15.78:0/2657940194 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75ac06b500 0x7f75ac158900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:14.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.278+0000 7f75ab7fe700 1 -- 172.21.15.78:0/2657940194 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a4009b30 con 0x7f75ac13dbe0 2023-10-14T11:59:14.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.278+0000 7f75aaffd700 1 --2- 172.21.15.78:0/2657940194 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75ac06b500 0x7f75ac158900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:59:14.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.278+0000 7f75aa7fc700 1 --2- 172.21.15.78:0/2657940194 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f75ac0c4540 0x7f75ac1564b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:59:14.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.279+0000 7f75ab7fe700 1 --2- 172.21.15.78:0/2657940194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75ac13dbe0 0x7f75ac13b640 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f75a4012000 tx=0x7f75a4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:14.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.279+0000 7f75937fe700 1 -- 172.21.15.78:0/2657940194 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a4004960 con 0x7f75ac13dbe0 2023-10-14T11:59:14.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.279+0000 7f75937fe700 1 -- 172.21.15.78:0/2657940194 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75a40099b0 con 0x7f75ac13dbe0 2023-10-14T11:59:14.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.279+0000 7f75b17c4700 1 -- 172.21.15.78:0/2657940194 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75ac155b90 con 0x7f75ac13dbe0 2023-10-14T11:59:14.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.279+0000 7f75937fe700 1 -- 172.21.15.78:0/2657940194 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a401b670 con 0x7f75ac13dbe0 2023-10-14T11:59:14.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.279+0000 7f75b17c4700 1 -- 172.21.15.78:0/2657940194 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75ac0a8770 con 0x7f75ac13dbe0 2023-10-14T11:59:14.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.280+0000 7f75937fe700 1 -- 172.21.15.78:0/2657940194 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f75a401b7d0 con 0x7f75ac13dbe0 2023-10-14T11:59:14.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.281+0000 7f75937fe700 1 --2- 172.21.15.78:0/2657940194 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f759806af00 0x7f759806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:14.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.281+0000 7f75937fe700 1 -- 172.21.15.78:0/2657940194 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f75a408ab60 con 0x7f75ac13dbe0 2023-10-14T11:59:14.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.281+0000 7f75aaffd700 1 --2- 172.21.15.78:0/2657940194 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f759806af00 0x7f759806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:14.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.281+0000 7f75aaffd700 1 --2- 172.21.15.78:0/2657940194 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f759806af00 0x7f759806d3c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f75ac0c30b0 tx=0x7f75a0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:14.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.283+0000 7f75bd9aeb80 1 Processor -- start 2023-10-14T11:59:14.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.283+0000 7f75bd9aeb80 1 -- start start 2023-10-14T11:59:14.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.284+0000 7f75bd9aeb80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x560ea8bd46b0 0x560ea8c12620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:14.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.284+0000 7f75bd9aeb80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ea8bd4a90 0x560ea8c12b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:14.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.284+0000 7f75bd9aeb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ea8bcec90 0x560ea8bf28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:14.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.284+0000 7f75bd9aeb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ea8b48c20 con 0x560ea8bcec90 2023-10-14T11:59:14.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.284+0000 7f75bd9aeb80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ea8b2dc10 con 0x560ea8bd4a90 2023-10-14T11:59:14.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.284+0000 7f75bd9aeb80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x560ea8b2b530 con 0x560ea8bd46b0 2023-10-14T11:59:14.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.284+0000 7f75ab7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ea8bcec90 0x560ea8bf28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:14.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.284+0000 7f75ab7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ea8bcec90 0x560ea8bf28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46512/0 (socket says 172.21.15.78:46512) 2023-10-14T11:59:14.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.284+0000 7f75ab7fe700 1 -- 172.21.15.78:0/1662944617 learned_addr learned my addr 172.21.15.78:0/1662944617 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:14.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.284+0000 7f75aa7fc700 1 --2- 172.21.15.78:0/1662944617 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ea8bd4a90 0x560ea8c12b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:14.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.284+0000 7f75aaffd700 1 --2- 172.21.15.78:0/1662944617 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x560ea8bd46b0 0x560ea8c12620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:14.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.284+0000 7f75ab7fe700 1 -- 172.21.15.78:0/1662944617 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x560ea8bd46b0 msgr2=0x560ea8c12620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:14.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.284+0000 7f75ab7fe700 1 --2- 172.21.15.78:0/1662944617 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x560ea8bd46b0 0x560ea8c12620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:14.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.284+0000 7f75ab7fe700 1 -- 172.21.15.78:0/1662944617 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ea8bd4a90 msgr2=0x560ea8c12b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:14.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.284+0000 7f75ab7fe700 1 --2- 172.21.15.78:0/1662944617 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ea8bd4a90 0x560ea8c12b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:14.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.284+0000 7f75ab7fe700 1 -- 172.21.15.78:0/1662944617 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ea8ae43a0 con 0x560ea8bcec90 2023-10-14T11:59:14.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.284+0000 7f75aa7fc700 1 --2- 172.21.15.78:0/1662944617 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ea8bd4a90 0x560ea8c12b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:59:14.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.284+0000 7f75ab7fe700 1 --2- 172.21.15.78:0/1662944617 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ea8bcec90 0x560ea8bf28a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f75a4055d20 tx=0x7f75a4059f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=398d4dc871a6d3ca server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:14.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.285+0000 7f7590ff9700 1 -- 172.21.15.78:0/1662944617 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a4061b20 con 0x560ea8bcec90 2023-10-14T11:59:14.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.285+0000 7f7590ff9700 1 -- 172.21.15.78:0/1662944617 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75a4003e10 con 0x560ea8bcec90 2023-10-14T11:59:14.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.285+0000 7f7590ff9700 1 -- 172.21.15.78:0/1662944617 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a40092e0 con 0x560ea8bcec90 2023-10-14T11:59:14.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.285+0000 7f75bd9aeb80 1 -- 172.21.15.78:0/1662944617 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ea8bcec90 msgr2=0x560ea8bf28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:14.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.285+0000 7f75bd9aeb80 1 --2- 172.21.15.78:0/1662944617 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ea8bcec90 0x560ea8bf28a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f75a4055d20 tx=0x7f75a4059f20 comp rx=0 tx=0).stop 2023-10-14T11:59:14.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.285+0000 7f75bd9aeb80 1 -- 172.21.15.78:0/1662944617 shutdown_connections 2023-10-14T11:59:14.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.285+0000 7f75bd9aeb80 1 --2- 172.21.15.78:0/1662944617 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ea8bcec90 0x560ea8bf28a0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:14.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.285+0000 7f75bd9aeb80 1 --2- 172.21.15.78:0/1662944617 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x560ea8bd46b0 0x560ea8c12620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:14.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.285+0000 7f75bd9aeb80 1 --2- 172.21.15.78:0/1662944617 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ea8bd4a90 0x560ea8c12b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:14.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.285+0000 7f75bd9aeb80 1 -- 172.21.15.78:0/1662944617 >> 172.21.15.78:0/1662944617 conn(0x560ea8b01d80 msgr2=0x560ea8bd23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:14.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.285+0000 7f75bd9aeb80 1 -- 172.21.15.78:0/1662944617 shutdown_connections 2023-10-14T11:59:14.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.285+0000 7f75bd9aeb80 1 -- 172.21.15.78:0/1662944617 wait complete. 2023-10-14T11:59:14.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.286+0000 7f75bd9aeb80 1 -- start start 2023-10-14T11:59:14.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.286+0000 7f75bd9aeb80 1 Processor -- start 2023-10-14T11:59:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.287+0000 7f75bd9aeb80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x560ea8bcec90 0x560ea8bf9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.287+0000 7f75bd9aeb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ea8bd46b0 0x560ea8c103a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:14.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.287+0000 7f75bd9aeb80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ea8bd4a90 0x560ea8c0ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:14.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.287+0000 7f75bd9aeb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ea8b26e20 con 0x560ea8bd46b0 2023-10-14T11:59:14.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.287+0000 7f75bd9aeb80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ea8ac36c0 con 0x560ea8bd4a90 2023-10-14T11:59:14.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.287+0000 7f75bd9aeb80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x560ea8ac3360 con 0x560ea8bcec90 2023-10-14T11:59:14.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.287+0000 7f75bd9aeb80 10 client.0 _ll_register_callbacks cb 0x560ea89207b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T11:59:14.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.287+0000 7f75aa7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ea8bd46b0 0x560ea8c103a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:14.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.287+0000 7f75aa7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ea8bd46b0 0x560ea8c103a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46516/0 (socket says 172.21.15.78:46516) 2023-10-14T11:59:14.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.287+0000 7f75aa7fc700 1 -- 172.21.15.78:0/3836911823 learned_addr learned my addr 172.21.15.78:0/3836911823 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:14.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.287+0000 7f75ab7fe700 1 --2- 172.21.15.78:0/3836911823 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ea8bd4a90 0x560ea8c0ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:14.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.287+0000 7f75aaffd700 1 --2- 172.21.15.78:0/3836911823 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x560ea8bcec90 0x560ea8bf9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:14.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.287+0000 7f75aa7fc700 1 -- 172.21.15.78:0/3836911823 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x560ea8bcec90 msgr2=0x560ea8bf9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:14.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.287+0000 7f75aa7fc700 1 --2- 172.21.15.78:0/3836911823 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x560ea8bcec90 0x560ea8bf9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:14.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.287+0000 7f75aa7fc700 1 -- 172.21.15.78:0/3836911823 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ea8bd4a90 msgr2=0x560ea8c0ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:14.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.287+0000 7f75aa7fc700 1 --2- 172.21.15.78:0/3836911823 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ea8bd4a90 0x560ea8c0ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:14.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.287+0000 7f75aa7fc700 1 -- 172.21.15.78:0/3836911823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75ac0a8770 con 0x560ea8bd46b0 2023-10-14T11:59:14.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.287+0000 7f75aa7fc700 1 --2- 172.21.15.78:0/3836911823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ea8bd46b0 0x560ea8c103a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f759c011310 tx=0x7f759c011620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65583837704656be server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:14.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.287+0000 7f7589ffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-14T11:59:14.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.287+0000 7f7589ffb700 1 -- 172.21.15.78:0/3836911823 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759c004230 con 0x560ea8bd46b0 2023-10-14T11:59:14.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.287+0000 7f7589ffb700 1 -- 172.21.15.78:0/3836911823 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f759c004390 con 0x560ea8bd46b0 2023-10-14T11:59:14.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.287+0000 7f75bd9aeb80 1 -- 172.21.15.78:0/3836911823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560ea8bc75b0 con 0x560ea8bd46b0 2023-10-14T11:59:14.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.288+0000 7f7589ffb700 1 -- 172.21.15.78:0/3836911823 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f759c004550 con 0x560ea8bd46b0 2023-10-14T11:59:14.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.288+0000 7f75bd9aeb80 10 client.5268 fetch_fsmap learned FSMap version 17 2023-10-14T11:59:14.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.288+0000 7f75bd9aeb80 1 -- 172.21.15.78:0/3836911823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x560ea8951d90 con 0x560ea8bd46b0 2023-10-14T11:59:14.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.288+0000 7f7589ffb700 1 -- 172.21.15.78:0/3836911823 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x560ea8bc75b0 con 0x560ea8bd46b0 2023-10-14T11:59:14.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.288+0000 7f7589ffb700 1 -- 172.21.15.78:0/3836911823 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 17) v1 ==== 3299+0+0 (secure 0 0 0) 0x7f759c0036a0 con 0x560ea8bd46b0 2023-10-14T11:59:14.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.288+0000 7f75bd9aeb80 10 client.5268 fetch_fsmap finished waiting for FSMap version 17 2023-10-14T11:59:14.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.288+0000 7f75bd9aeb80 10 client.5268 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-14T11:59:14.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.288+0000 7f75bd9aeb80 20 client.5268 populate_metadata read hostname 'smithi078' 2023-10-14T11:59:14.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.288+0000 7f75bd9aeb80 1 --2- 172.21.15.78:0/3836911823 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x560ea8c23190 0x560ea8c23570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:14.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.288+0000 7f75bd9aeb80 4 client.5268 mds_command: new command op to 5031 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-14T11:59:14.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.288+0000 7f75bd9aeb80 1 -- 172.21.15.78:0/3836911823 --> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560ea8b1fe00 con 0x560ea8c23190 2023-10-14T11:59:14.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.288+0000 7f75aaffd700 1 --2- 172.21.15.78:0/3836911823 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x560ea8c23190 0x560ea8c23570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:14.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.288+0000 7f7589ffb700 10 client.5268 ms_handle_connect on v2:172.21.15.195:6833/2130671508 2023-10-14T11:59:14.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.288+0000 7f75aaffd700 1 --2- 172.21.15.78:0/3836911823 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x560ea8c23190 0x560ea8c23570 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa301917eec25bb4 server_cookie=4dd824111af0765c in_seq=0 out_seq=0 2023-10-14T11:59:14.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.289+0000 7f7589ffb700 1 -- 172.21.15.78:0/3836911823 <== mds.0 v2:172.21.15.195:6833/2130671508 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x560ea8b1fe00 con 0x560ea8c23190 2023-10-14T11:59:14.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.289+0000 7f7589ffb700 10 client.5268 handle_command_reply: tid=0 2023-10-14T11:59:14.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.289+0000 7f75bd9aeb80 1 client.5268 shutdown 2023-10-14T11:59:14.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.289+0000 7f75bd9aeb80 1 -- 172.21.15.78:0/3836911823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ea8bd46b0 msgr2=0x560ea8c103a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:14.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.289+0000 7f75bd9aeb80 1 --2- 172.21.15.78:0/3836911823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ea8bd46b0 0x560ea8c103a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f759c011310 tx=0x7f759c011620 comp rx=0 tx=0).stop 2023-10-14T11:59:14.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.289+0000 7f75bd9aeb80 1 -- 172.21.15.78:0/3836911823 shutdown_connections 2023-10-14T11:59:14.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.289+0000 7f75bd9aeb80 1 --2- 172.21.15.78:0/3836911823 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x560ea8c23190 0x560ea8c23570 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:14.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.289+0000 7f75bd9aeb80 1 --2- 172.21.15.78:0/3836911823 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ea8bd4a90 0x560ea8c0ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:14.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.289+0000 7f75bd9aeb80 1 --2- 172.21.15.78:0/3836911823 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x560ea8bcec90 0x560ea8bf9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:14.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.289+0000 7f7589ffb700 0 client.5268 ms_handle_reset on v2:172.21.15.195:6833/2130671508 2023-10-14T11:59:14.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.289+0000 7f75bd9aeb80 1 --2- 172.21.15.78:0/3836911823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ea8bd46b0 0x560ea8c103a0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:14.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.289+0000 7f75bd9aeb80 1 -- 172.21.15.78:0/3836911823 >> 172.21.15.78:0/3836911823 conn(0x560ea8b01d80 msgr2=0x560ea8bc97d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:14.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.289+0000 7f75bd9aeb80 1 -- 172.21.15.78:0/3836911823 shutdown_connections 2023-10-14T11:59:14.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.289+0000 7f75bd9aeb80 1 -- 172.21.15.78:0/3836911823 wait complete. 2023-10-14T11:59:14.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.289+0000 7f75bd9aeb80 20 client.5268 trim_cache size 0 max 16384 2023-10-14T11:59:14.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.307+0000 7f75bd9aeb80 1 Processor -- start 2023-10-14T11:59:14.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.307+0000 7f75bd9aeb80 1 -- start start 2023-10-14T11:59:14.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.307+0000 7f75bd9aeb80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ea8bcec90 0x560ea8bf4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:14.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.307+0000 7f75bd9aeb80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x560ea8bd46b0 0x560ea8bf8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:14.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.307+0000 7f75bd9aeb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ea8bd4a90 0x560ea8bf93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:14.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.307+0000 7f75bd9aeb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ea89a78d0 con 0x560ea8bd4a90 2023-10-14T11:59:14.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.307+0000 7f75bd9aeb80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ea8c20640 con 0x560ea8bcec90 2023-10-14T11:59:14.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.307+0000 7f75bd9aeb80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x560ea8c10c00 con 0x560ea8bd46b0 2023-10-14T11:59:14.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.308+0000 7f75ab7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ea8bd4a90 0x560ea8bf93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:14.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.308+0000 7f75ab7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ea8bd4a90 0x560ea8bf93d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46526/0 (socket says 172.21.15.78:46526) 2023-10-14T11:59:14.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.308+0000 7f75ab7fe700 1 -- 172.21.15.78:0/2067189335 learned_addr learned my addr 172.21.15.78:0/2067189335 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:14.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.308+0000 7f75aa7fc700 1 --2- 172.21.15.78:0/2067189335 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x560ea8bd46b0 0x560ea8bf8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:14.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.308+0000 7f75aaffd700 1 --2- 172.21.15.78:0/2067189335 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ea8bcec90 0x560ea8bf4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:14.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.308+0000 7f75ab7fe700 1 -- 172.21.15.78:0/2067189335 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x560ea8bd46b0 msgr2=0x560ea8bf8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:14.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.308+0000 7f75ab7fe700 1 --2- 172.21.15.78:0/2067189335 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x560ea8bd46b0 0x560ea8bf8e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:14.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.308+0000 7f75ab7fe700 1 -- 172.21.15.78:0/2067189335 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ea8bcec90 msgr2=0x560ea8bf4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:14.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.308+0000 7f75ab7fe700 1 --2- 172.21.15.78:0/2067189335 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ea8bcec90 0x560ea8bf4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:14.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.308+0000 7f75ab7fe700 1 -- 172.21.15.78:0/2067189335 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ea893ac10 con 0x560ea8bd4a90 2023-10-14T11:59:14.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.308+0000 7f75ab7fe700 1 --2- 172.21.15.78:0/2067189335 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ea8bd4a90 0x560ea8bf93d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f75a4056240 tx=0x7f75a401f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2866c7ea4638080 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:14.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.308+0000 7f75917fa700 1 -- 172.21.15.78:0/2067189335 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a4055b60 con 0x560ea8bd4a90 2023-10-14T11:59:14.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.308+0000 7f75917fa700 1 -- 172.21.15.78:0/2067189335 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75a4071110 con 0x560ea8bd4a90 2023-10-14T11:59:14.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.308+0000 7f75917fa700 1 -- 172.21.15.78:0/2067189335 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a40092e0 con 0x560ea8bd4a90 2023-10-14T11:59:14.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.309+0000 7f75bd9aeb80 1 -- 172.21.15.78:0/2067189335 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ea8bd4a90 msgr2=0x560ea8bf93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:14.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.309+0000 7f75bd9aeb80 1 --2- 172.21.15.78:0/2067189335 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ea8bd4a90 0x560ea8bf93d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f75a4056240 tx=0x7f75a401f000 comp rx=0 tx=0).stop 2023-10-14T11:59:14.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.309+0000 7f75bd9aeb80 1 -- 172.21.15.78:0/2067189335 shutdown_connections 2023-10-14T11:59:14.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.309+0000 7f75bd9aeb80 1 --2- 172.21.15.78:0/2067189335 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ea8bd4a90 0x560ea8bf93d0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:14.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.309+0000 7f75bd9aeb80 1 --2- 172.21.15.78:0/2067189335 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ea8bcec90 0x560ea8bf4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:14.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.309+0000 7f75bd9aeb80 1 --2- 172.21.15.78:0/2067189335 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x560ea8bd46b0 0x560ea8bf8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:14.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.309+0000 7f75bd9aeb80 1 -- 172.21.15.78:0/2067189335 >> 172.21.15.78:0/2067189335 conn(0x560ea8b01d80 msgr2=0x560ea8c327f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:14.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.309+0000 7f75bd9aeb80 1 -- 172.21.15.78:0/2067189335 shutdown_connections 2023-10-14T11:59:14.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.309+0000 7f75bd9aeb80 1 -- 172.21.15.78:0/2067189335 wait complete. 2023-10-14T11:59:14.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.309+0000 7f75bd9aeb80 1 -- start start 2023-10-14T11:59:14.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.309+0000 7f75bd9aeb80 1 Processor -- start 2023-10-14T11:59:14.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.310+0000 7f75bd9aeb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ea8bcec90 0x560ea8bf0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:14.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.310+0000 7f75bd9aeb80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ea8bd46b0 0x560ea8c779f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:14.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.310+0000 7f75bd9aeb80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x560ea8bd4a90 0x560ea8c749b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:14.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.310+0000 7f75bd9aeb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ea8c29920 con 0x560ea8bcec90 2023-10-14T11:59:14.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.310+0000 7f75bd9aeb80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ea8c775a0 con 0x560ea8bd46b0 2023-10-14T11:59:14.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.310+0000 7f75bd9aeb80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x560ea8be9660 con 0x560ea8bd4a90 2023-10-14T11:59:14.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.310+0000 7f75bd9aeb80 10 client.0 _ll_register_callbacks cb 0x7f759c009b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T11:59:14.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.310+0000 7f75aaffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ea8bcec90 0x560ea8bf0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:14.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.310+0000 7f75aaffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ea8bcec90 0x560ea8bf0a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46528/0 (socket says 172.21.15.78:46528) 2023-10-14T11:59:14.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.310+0000 7f75aaffd700 1 -- 172.21.15.78:0/523976378 learned_addr learned my addr 172.21.15.78:0/523976378 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:14.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.310+0000 7f75ab7fe700 1 --2- 172.21.15.78:0/523976378 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x560ea8bd4a90 0x560ea8c749b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:14.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.310+0000 7f75aa7fc700 1 --2- 172.21.15.78:0/523976378 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ea8bd46b0 0x560ea8c779f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:14.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.310+0000 7f75aaffd700 1 -- 172.21.15.78:0/523976378 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x560ea8bd4a90 msgr2=0x560ea8c749b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:14.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.310+0000 7f75aaffd700 1 --2- 172.21.15.78:0/523976378 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x560ea8bd4a90 0x560ea8c749b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:14.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.310+0000 7f75aaffd700 1 -- 172.21.15.78:0/523976378 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ea8bd46b0 msgr2=0x560ea8c779f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:14.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.310+0000 7f75aaffd700 1 --2- 172.21.15.78:0/523976378 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ea8bd46b0 0x560ea8c779f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:14.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.310+0000 7f75aaffd700 1 -- 172.21.15.78:0/523976378 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560ea8ae43a0 con 0x560ea8bcec90 2023-10-14T11:59:14.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.310+0000 7f75ab7fe700 1 --2- 172.21.15.78:0/523976378 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x560ea8bd4a90 0x560ea8c749b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:59:14.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.310+0000 7f75aaffd700 1 --2- 172.21.15.78:0/523976378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ea8bcec90 0x560ea8bf0a10 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f75a0007fd0 tx=0x7f75a0000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=726fa96937f9b2a0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:14.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.310+0000 7f7589ffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-14T11:59:14.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.311+0000 7f7589ffb700 1 -- 172.21.15.78:0/523976378 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a0029070 con 0x560ea8bcec90 2023-10-14T11:59:14.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.311+0000 7f7589ffb700 1 -- 172.21.15.78:0/523976378 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f75a0003970 con 0x560ea8bcec90 2023-10-14T11:59:14.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.311+0000 7f75bd9aeb80 1 -- 172.21.15.78:0/523976378 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f75a4003e10 con 0x560ea8bcec90 2023-10-14T11:59:14.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.311+0000 7f7589ffb700 1 -- 172.21.15.78:0/523976378 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f75a0003b30 con 0x560ea8bcec90 2023-10-14T11:59:14.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.311+0000 7f75bd9aeb80 10 client.5274 fetch_fsmap learned FSMap version 17 2023-10-14T11:59:14.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.311+0000 7f75bd9aeb80 1 -- 172.21.15.78:0/523976378 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x560ea8978200 con 0x560ea8bcec90 2023-10-14T11:59:14.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.311+0000 7f7589ffb700 1 -- 172.21.15.78:0/523976378 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75a4003e10 con 0x560ea8bcec90 2023-10-14T11:59:14.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.311+0000 7f7589ffb700 1 -- 172.21.15.78:0/523976378 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 17) v1 ==== 3299+0+0 (secure 0 0 0) 0x7f75a0003c90 con 0x560ea8bcec90 2023-10-14T11:59:14.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.311+0000 7f75bd9aeb80 10 client.5274 fetch_fsmap finished waiting for FSMap version 17 2023-10-14T11:59:14.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.311+0000 7f75bd9aeb80 10 client.5274 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-14T11:59:14.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.311+0000 7f75bd9aeb80 20 client.5274 populate_metadata read hostname 'smithi078' 2023-10-14T11:59:14.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.311+0000 7f75bd9aeb80 1 --2- 172.21.15.78:0/523976378 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x560ea8c7ab10 0x560ea8c7aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:14.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.311+0000 7f75bd9aeb80 4 client.5274 mds_command: new command op to 5031 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-14T11:59:14.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.311+0000 7f75bd9aeb80 1 -- 172.21.15.78:0/523976378 --> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560ea8b1fb20 con 0x560ea8c7ab10 2023-10-14T11:59:14.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.311+0000 7f75aa7fc700 1 --2- 172.21.15.78:0/523976378 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x560ea8c7ab10 0x560ea8c7aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:14.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.312+0000 7f7589ffb700 10 client.5274 ms_handle_connect on v2:172.21.15.195:6833/2130671508 2023-10-14T11:59:14.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.312+0000 7f75aa7fc700 1 --2- 172.21.15.78:0/523976378 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x560ea8c7ab10 0x560ea8c7aef0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b98384d7ed1a583 server_cookie=98da2746d12b17db in_seq=0 out_seq=0 2023-10-14T11:59:14.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.312+0000 7f7589ffb700 1 -- 172.21.15.78:0/523976378 <== mds.0 v2:172.21.15.195:6833/2130671508 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+336 (crc 0 0 0) 0x560ea8b1fb20 con 0x560ea8c7ab10 2023-10-14T11:59:14.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.312+0000 7f7589ffb700 10 client.5274 handle_command_reply: tid=0 2023-10-14T11:59:14.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.312+0000 7f75bd9aeb80 1 client.5274 shutdown 2023-10-14T11:59:14.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.312+0000 7f75bd9aeb80 1 -- 172.21.15.78:0/523976378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ea8bcec90 msgr2=0x560ea8bf0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:14.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.312+0000 7f75bd9aeb80 1 --2- 172.21.15.78:0/523976378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ea8bcec90 0x560ea8bf0a10 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f75a0007fd0 tx=0x7f75a0000ed0 comp rx=0 tx=0).stop 2023-10-14T11:59:14.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.312+0000 7f75bd9aeb80 1 -- 172.21.15.78:0/523976378 shutdown_connections 2023-10-14T11:59:14.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.312+0000 7f75bd9aeb80 1 --2- 172.21.15.78:0/523976378 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x560ea8bd4a90 0x560ea8c749b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:14.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.312+0000 7f75bd9aeb80 1 --2- 172.21.15.78:0/523976378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ea8bcec90 0x560ea8bf0a10 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:14.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.312+0000 7f75bd9aeb80 1 --2- 172.21.15.78:0/523976378 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x560ea8c7ab10 0x560ea8c7aef0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:14.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.312+0000 7f75bd9aeb80 1 --2- 172.21.15.78:0/523976378 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ea8bd46b0 0x560ea8c779f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:14.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.312+0000 7f7589ffb700 0 client.5274 ms_handle_reset on v2:172.21.15.195:6833/2130671508 2023-10-14T11:59:14.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.312+0000 7f75bd9aeb80 1 -- 172.21.15.78:0/523976378 >> 172.21.15.78:0/523976378 conn(0x560ea8b01d80 msgr2=0x560ea8bf44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:14.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.312+0000 7f75bd9aeb80 1 -- 172.21.15.78:0/523976378 shutdown_connections 2023-10-14T11:59:14.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.312+0000 7f75bd9aeb80 1 -- 172.21.15.78:0/523976378 wait complete. 2023-10-14T11:59:14.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.312+0000 7f75bd9aeb80 20 client.5274 trim_cache size 0 max 16384 2023-10-14T11:59:14.340 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-14T11:59:14.340 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "4c5dc134-3092-406a-a1e1-13e6d9e926e8", 2023-10-14T11:59:14.340 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2023-10-14T11:59:14.340 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 5031, 2023-10-14T11:59:14.341 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2023-10-14T11:59:14.341 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2023-10-14T11:59:14.341 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "testcephfs1", 2023-10-14T11:59:14.341 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 12.128017463999999, 2023-10-14T11:59:14.341 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 17, 2023-10-14T11:59:14.342 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 32, 2023-10-14T11:59:14.342 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 32, 2023-10-14T11:59:14.342 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 20.018841397999999 2023-10-14T11:59:14.342 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-14T11:59:14.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.313+0000 7f75b17c4700 1 -- 172.21.15.78:0/2657940194 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f759806af00 msgr2=0x7f759806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:14.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.313+0000 7f75b17c4700 1 --2- 172.21.15.78:0/2657940194 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f759806af00 0x7f759806d3c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f75ac0c30b0 tx=0x7f75a0009040 comp rx=0 tx=0).stop 2023-10-14T11:59:14.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.313+0000 7f75b17c4700 1 -- 172.21.15.78:0/2657940194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75ac13dbe0 msgr2=0x7f75ac13b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:14.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.313+0000 7f75b17c4700 1 --2- 172.21.15.78:0/2657940194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75ac13dbe0 0x7f75ac13b640 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f75a4012000 tx=0x7f75a4004ad0 comp rx=0 tx=0).stop 2023-10-14T11:59:14.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.313+0000 7f75b17c4700 1 -- 172.21.15.78:0/2657940194 shutdown_connections 2023-10-14T11:59:14.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.313+0000 7f75b17c4700 1 --2- 172.21.15.78:0/2657940194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75ac13dbe0 0x7f75ac13b640 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:14.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.313+0000 7f75b17c4700 1 --2- 172.21.15.78:0/2657940194 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f759806af00 0x7f759806d3c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:14.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.313+0000 7f75b17c4700 1 --2- 172.21.15.78:0/2657940194 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75ac06b500 0x7f75ac158900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:14.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.313+0000 7f75b17c4700 1 --2- 172.21.15.78:0/2657940194 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f75ac0c4540 0x7f75ac1564b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:14.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.313+0000 7f75b17c4700 1 -- 172.21.15.78:0/2657940194 >> 172.21.15.78:0/2657940194 conn(0x7f75ac0c3a50 msgr2=0x7f75ac0c2480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:14.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.313+0000 7f75b17c4700 1 -- 172.21.15.78:0/2657940194 shutdown_connections 2023-10-14T11:59:14.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:14.313+0000 7f75b17c4700 1 -- 172.21.15.78:0/2657940194 wait complete. 2023-10-14T11:59:14.346 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 17, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-14T11:59:01.562764+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 17, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "testcephfs1", "in": [ 0 ], "info": { "gid_5031": { "addr": "172.21.15.195:6835/2130671508", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6833", "nonce": 2130671508, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2130671508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5031, "incarnation": 13, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2023-10-14T11:59:04.592935+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5031 } } } ], "standbys": [ { "addr": "172.21.15.195:6834/1492486786", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6832", "nonce": 1492486786, "type": "v2" }, { "addr": "172.21.15.195:6834", "nonce": 1492486786, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4658, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6836/2991384297", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6834", "nonce": 2991384297, "type": "v2" }, { "addr": "172.21.15.78:6836", "nonce": 2991384297, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5016, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6837/3246640037", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6835", "nonce": 3246640037, "type": "v2" }, { "addr": "172.21.15.78:6837", "nonce": 3246640037, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5019, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-14T11:59:14.348 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 2, 'name': 'b', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2023-10-14T11:59:14.348 INFO:tasks.cephfs.test_failover:current = [{'gid': 5019, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6837/3246640037', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 3246640037}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 3246640037}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 4658, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.195:6834/1492486786', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6832', 'nonce': 1492486786}, {'type': 'v1', 'addr': '172.21.15.195:6834', 'nonce': 1492486786}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 5016, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6836/2991384297', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 2991384297}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 2991384297}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 5031, 'name': 'd', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.195:6835/2130671508', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6833', 'nonce': 2130671508}, {'type': 'v1', 'addr': '172.21.15.195:6835', 'nonce': 2130671508}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2023-10-14T11:59:14.349 DEBUG:tasks.cephfs.test_failover:2 != -1 2023-10-14T11:59:14.349 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2023-10-14T11:59:19.351 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T11:59:19.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.487+0000 7ff35f5fd700 1 Processor -- start 2023-10-14T11:59:19.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.488+0000 7ff35f5fd700 1 -- start start 2023-10-14T11:59:19.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.488+0000 7ff35f5fd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff35806f1e0 0x7ff3581554e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:19.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.488+0000 7ff35f5fd700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff3580c0350 0x7ff35813d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:19.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.489+0000 7ff35f5fd700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff35806b630 con 0x7ff35806ab40 2023-10-14T11:59:19.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.489+0000 7ff35f5fd700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3580c49f0 con 0x7ff35806f1e0 2023-10-14T11:59:19.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.489+0000 7ff35f5fd700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3580726a0 con 0x7ff3580c0350 2023-10-14T11:59:19.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.489+0000 7ff35cb98700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff35806f1e0 0x7ff3581554e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:19.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.489+0000 7ff35db9a700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff3580c0350 0x7ff35813d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:19.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.489+0000 7ff35cb98700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff35806f1e0 0x7ff3581554e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.78:49276/0 (socket says 172.21.15.78:49276) 2023-10-14T11:59:19.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.489+0000 7ff35db9a700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff3580c0350 0x7ff35813d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.78:46210/0 (socket says 172.21.15.78:46210) 2023-10-14T11:59:19.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.489+0000 7ff35cb98700 1 -- 172.21.15.78:0/535664624 learned_addr learned my addr 172.21.15.78:0/535664624 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:19.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.489+0000 7ff35db9a700 1 -- 172.21.15.78:0/535664624 >> v2:172.21.15.195:3300/0 conn(0x7ff35806f1e0 msgr2=0x7ff3581554e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:19.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.489+0000 7ff35db9a700 1 --2- 172.21.15.78:0/535664624 >> v2:172.21.15.195:3300/0 conn(0x7ff35806f1e0 0x7ff3581554e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:19.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.489+0000 7ff35db9a700 1 -- 172.21.15.78:0/535664624 >> v1:172.21.15.195:6789/0 conn(0x7ff35806ab40 legacy=0x7ff35806af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:19.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.489+0000 7ff35db9a700 1 -- 172.21.15.78:0/535664624 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff358072980 con 0x7ff3580c0350 2023-10-14T11:59:19.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.489+0000 7ff35db9a700 1 --2- 172.21.15.78:0/535664624 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff3580c0350 0x7ff35813d7b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff350009d90 tx=0x7ff35000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a21089d24f510021 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:19.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.490+0000 7ff34ffff700 1 -- 172.21.15.78:0/535664624 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff35000e070 con 0x7ff3580c0350 2023-10-14T11:59:19.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.490+0000 7ff34ffff700 1 -- 172.21.15.78:0/535664624 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff35000bb20 con 0x7ff3580c0350 2023-10-14T11:59:19.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.490+0000 7ff34ffff700 1 -- 172.21.15.78:0/535664624 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff350012430 con 0x7ff3580c0350 2023-10-14T11:59:19.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.490+0000 7ff35f5fd700 1 -- 172.21.15.78:0/535664624 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff3580c0350 msgr2=0x7ff35813d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:19.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.490+0000 7ff35f5fd700 1 --2- 172.21.15.78:0/535664624 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff3580c0350 0x7ff35813d7b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff350009d90 tx=0x7ff35000b330 comp rx=0 tx=0).stop 2023-10-14T11:59:19.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.490+0000 7ff35f5fd700 1 -- 172.21.15.78:0/535664624 shutdown_connections 2023-10-14T11:59:19.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.490+0000 7ff35f5fd700 1 --2- 172.21.15.78:0/535664624 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff3580c0350 0x7ff35813d7b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:19.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.490+0000 7ff35f5fd700 1 --2- 172.21.15.78:0/535664624 >> v2:172.21.15.195:3300/0 conn(0x7ff35806f1e0 0x7ff3581554e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:19.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.490+0000 7ff35f5fd700 1 -- 172.21.15.78:0/535664624 >> 172.21.15.78:0/535664624 conn(0x7ff3580c3ac0 msgr2=0x7ff3580c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:19.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.490+0000 7ff35f5fd700 1 -- 172.21.15.78:0/535664624 shutdown_connections 2023-10-14T11:59:19.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.490+0000 7ff35f5fd700 1 -- 172.21.15.78:0/535664624 wait complete. 2023-10-14T11:59:19.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.491+0000 7ff35f5fd700 1 Processor -- start 2023-10-14T11:59:19.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.492+0000 7ff35f5fd700 1 -- start start 2023-10-14T11:59:19.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.492+0000 7ff35f5fd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff35806ab40 0x7ff3581ecd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:19.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.492+0000 7ff35f5fd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff35806f1e0 0x7ff3580aacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:19.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.492+0000 7ff35f5fd700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff3580c0350 0x7ff3580ab200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:19.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.492+0000 7ff35f5fd700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff358158280 con 0x7ff35806ab40 2023-10-14T11:59:19.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.492+0000 7ff35f5fd700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35814f510 con 0x7ff35806f1e0 2023-10-14T11:59:19.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.492+0000 7ff35f5fd700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff358151d80 con 0x7ff3580c0350 2023-10-14T11:59:19.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.492+0000 7ff35d399700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff35806ab40 0x7ff3581ecd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:19.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.492+0000 7ff35d399700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff35806ab40 0x7ff3581ecd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46542/0 (socket says 172.21.15.78:46542) 2023-10-14T11:59:19.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.492+0000 7ff35d399700 1 -- 172.21.15.78:0/2780541826 learned_addr learned my addr 172.21.15.78:0/2780541826 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:19.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.493+0000 7ff35cb98700 1 --2- 172.21.15.78:0/2780541826 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff35806f1e0 0x7ff3580aacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:19.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.493+0000 7ff35db9a700 1 --2- 172.21.15.78:0/2780541826 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff3580c0350 0x7ff3580ab200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:19.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.493+0000 7ff35d399700 1 -- 172.21.15.78:0/2780541826 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff3580c0350 msgr2=0x7ff3580ab200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:19.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.493+0000 7ff35d399700 1 --2- 172.21.15.78:0/2780541826 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff3580c0350 0x7ff3580ab200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:19.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.493+0000 7ff35d399700 1 -- 172.21.15.78:0/2780541826 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff35806f1e0 msgr2=0x7ff3580aacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:19.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.493+0000 7ff35d399700 1 --2- 172.21.15.78:0/2780541826 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff35806f1e0 0x7ff3580aacc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:19.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.493+0000 7ff35d399700 1 -- 172.21.15.78:0/2780541826 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff350009a70 con 0x7ff35806ab40 2023-10-14T11:59:19.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.493+0000 7ff35d399700 1 --2- 172.21.15.78:0/2780541826 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff35806ab40 0x7ff3581ecd90 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff34800c450 tx=0x7ff34800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:19.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.493+0000 7ff34dffb700 1 -- 172.21.15.78:0/2780541826 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3480052d0 con 0x7ff35806ab40 2023-10-14T11:59:19.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.493+0000 7ff34dffb700 1 -- 172.21.15.78:0/2780541826 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff348005430 con 0x7ff35806ab40 2023-10-14T11:59:19.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.493+0000 7ff35f5fd700 1 -- 172.21.15.78:0/2780541826 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3581ed810 con 0x7ff35806ab40 2023-10-14T11:59:19.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.493+0000 7ff34dffb700 1 -- 172.21.15.78:0/2780541826 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3480038c0 con 0x7ff35806ab40 2023-10-14T11:59:19.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.493+0000 7ff35f5fd700 1 -- 172.21.15.78:0/2780541826 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3580c6ad0 con 0x7ff35806ab40 2023-10-14T11:59:19.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.494+0000 7ff35f5fd700 1 -- 172.21.15.78:0/2780541826 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff35814ef90 con 0x7ff35806ab40 2023-10-14T11:59:19.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.494+0000 7ff34dffb700 1 -- 172.21.15.78:0/2780541826 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff348003a20 con 0x7ff35806ab40 2023-10-14T11:59:19.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.495+0000 7ff34dffb700 1 --2- 172.21.15.78:0/2780541826 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff33c06af00 0x7ff33c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:19.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.495+0000 7ff34dffb700 1 -- 172.21.15.78:0/2780541826 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7ff3480888c0 con 0x7ff35806ab40 2023-10-14T11:59:19.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.495+0000 7ff35cb98700 1 --2- 172.21.15.78:0/2780541826 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff33c06af00 0x7ff33c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:19.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.495+0000 7ff35cb98700 1 --2- 172.21.15.78:0/2780541826 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff33c06af00 0x7ff33c06d3c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff358152100 tx=0x7ff354008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:19.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.500+0000 7ff34dffb700 1 -- 172.21.15.78:0/2780541826 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff35814ef90 con 0x7ff35806ab40 2023-10-14T11:59:19.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.730+0000 7ff35f5fd700 1 -- 172.21.15.78:0/2780541826 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff35814ef90 con 0x7ff35806ab40 2023-10-14T11:59:19.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.731+0000 7ff34dffb700 1 -- 172.21.15.78:0/2780541826 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4924 (secure 0 0 0) 0x7ff35814ef90 con 0x7ff35806ab40 2023-10-14T11:59:19.733 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:59:19.733 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4658,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6834/1492486786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1492486786},{"type":"v1","addr":"172.21.15.195:6834","nonce":1492486786}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5016,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/2991384297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2991384297},{"type":"v1","addr":"172.21.15.78:6836","nonce":2991384297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5019,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3246640037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3246640037},{"type":"v1","addr":"172.21.15.78:6837","nonce":3246640037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T11:59:01.562764+0000","modified":"2023-10-14T11:59:04.592935+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5031":{"gid":5031,"name":"d","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.195:6835/2130671508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6833","nonce":2130671508},{"type":"v1","addr":"172.21.15.195:6835","nonce":2130671508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"testcephfs1","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-14T11:59:19.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.733+0000 7ff35f5fd700 1 -- 172.21.15.78:0/2780541826 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff33c06af00 msgr2=0x7ff33c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:19.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.733+0000 7ff35f5fd700 1 --2- 172.21.15.78:0/2780541826 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff33c06af00 0x7ff33c06d3c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff358152100 tx=0x7ff354008040 comp rx=0 tx=0).stop 2023-10-14T11:59:19.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.733+0000 7ff35f5fd700 1 -- 172.21.15.78:0/2780541826 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff35806ab40 msgr2=0x7ff3581ecd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:19.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.733+0000 7ff35f5fd700 1 --2- 172.21.15.78:0/2780541826 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff35806ab40 0x7ff3581ecd90 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff34800c450 tx=0x7ff34800c810 comp rx=0 tx=0).stop 2023-10-14T11:59:19.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.733+0000 7ff35f5fd700 1 -- 172.21.15.78:0/2780541826 shutdown_connections 2023-10-14T11:59:19.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.733+0000 7ff35f5fd700 1 --2- 172.21.15.78:0/2780541826 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff3580c0350 0x7ff3580ab200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:19.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.733+0000 7ff35f5fd700 1 --2- 172.21.15.78:0/2780541826 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff35806ab40 0x7ff3581ecd90 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:19.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.733+0000 7ff35f5fd700 1 --2- 172.21.15.78:0/2780541826 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff33c06af00 0x7ff33c06d3c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:19.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.733+0000 7ff35f5fd700 1 --2- 172.21.15.78:0/2780541826 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff35806f1e0 0x7ff3580aacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:19.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.733+0000 7ff35f5fd700 1 -- 172.21.15.78:0/2780541826 >> 172.21.15.78:0/2780541826 conn(0x7ff3580c3ac0 msgr2=0x7ff358153300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:19.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.733+0000 7ff35f5fd700 1 -- 172.21.15.78:0/2780541826 shutdown_connections 2023-10-14T11:59:19.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:19.733+0000 7ff35f5fd700 1 -- 172.21.15.78:0/2780541826 wait complete. 2023-10-14T11:59:19.740 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 17 2023-10-14T11:59:20.150 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-14T11:59:01.562764+0000', 'modified': '2023-10-14T11:59:04.592935+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5031}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5031': {'gid': 5031, 'name': 'd', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.195:6835/2130671508', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6833', 'nonce': 2130671508}, {'type': 'v1', 'addr': '172.21.15.195:6835', 'nonce': 2130671508}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'testcephfs1', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-14T11:59:20.150 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-14T11:59:20.151 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-14T11:59:20.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.273+0000 7f2b97240700 1 Processor -- start 2023-10-14T11:59:20.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.273+0000 7f2b97240700 1 -- start start 2023-10-14T11:59:20.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.274+0000 7f2b97240700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f2b9006f430 0x7f2b9006f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:20.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.274+0000 7f2b97240700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2b90155e00 0x7f2b90156290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:20.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.274+0000 7f2b97240700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b90152850 con 0x7f2b90072390 2023-10-14T11:59:20.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.274+0000 7f2b97240700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b9013bf20 con 0x7f2b90155e00 2023-10-14T11:59:20.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.274+0000 7f2b97240700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b90152b10 con 0x7f2b9006f430 2023-10-14T11:59:20.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.274+0000 7f2b957dd700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2b90155e00 0x7f2b90156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:20.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.274+0000 7f2b8ffff700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f2b90072390 0x7f2b9013b810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51430/0 (socket says 172.21.15.78:51430) 2023-10-14T11:59:20.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.274+0000 7f2b8ffff700 1 -- 172.21.15.78:0/2326616350 learned_addr learned my addr 172.21.15.78:0/2326616350 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:59:20.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.274+0000 7f2b94fdc700 1 --2- 172.21.15.78:0/2326616350 >> v2:172.21.15.195:3300/0 conn(0x7f2b9006f430 0x7f2b9006f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:20.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.274+0000 7f2b957dd700 1 -- 172.21.15.78:0/2326616350 >> v2:172.21.15.195:3300/0 conn(0x7f2b9006f430 msgr2=0x7f2b9006f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:20.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.274+0000 7f2b957dd700 1 --2- 172.21.15.78:0/2326616350 >> v2:172.21.15.195:3300/0 conn(0x7f2b9006f430 0x7f2b9006f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:20.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.274+0000 7f2b957dd700 1 -- 172.21.15.78:0/2326616350 >> v1:172.21.15.78:6789/0 conn(0x7f2b90072390 legacy=0x7f2b9013b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:20.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.274+0000 7f2b957dd700 1 -- 172.21.15.78:0/2326616350 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b90152df0 con 0x7f2b90155e00 2023-10-14T11:59:20.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.274+0000 7f2b8f7fe700 1 -- 172.21.15.78:0/2326616350 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1296513178 0 0) 0x7f2b90152850 con 0x7f2b90072390 2023-10-14T11:59:20.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.275+0000 7f2b957dd700 1 --2- 172.21.15.78:0/2326616350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b90155e00 0x7f2b90156290 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f2b90072b80 tx=0x7f2b8800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=784d75b7e9a2f2d5 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:20.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.275+0000 7f2b8f7fe700 1 -- 172.21.15.78:0/2326616350 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8800e070 con 0x7f2b90155e00 2023-10-14T11:59:20.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.275+0000 7f2b8f7fe700 1 -- 172.21.15.78:0/2326616350 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b8800bbd0 con 0x7f2b90155e00 2023-10-14T11:59:20.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.275+0000 7f2b8f7fe700 1 -- 172.21.15.78:0/2326616350 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b88012430 con 0x7f2b90155e00 2023-10-14T11:59:20.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.275+0000 7f2b97240700 1 -- 172.21.15.78:0/2326616350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b90155e00 msgr2=0x7f2b90156290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:20.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.275+0000 7f2b97240700 1 --2- 172.21.15.78:0/2326616350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b90155e00 0x7f2b90156290 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f2b90072b80 tx=0x7f2b8800b330 comp rx=0 tx=0).stop 2023-10-14T11:59:20.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.275+0000 7f2b97240700 1 -- 172.21.15.78:0/2326616350 shutdown_connections 2023-10-14T11:59:20.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.275+0000 7f2b97240700 1 --2- 172.21.15.78:0/2326616350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b90155e00 0x7f2b90156290 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:20.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.275+0000 7f2b97240700 1 --2- 172.21.15.78:0/2326616350 >> v2:172.21.15.195:3300/0 conn(0x7f2b9006f430 0x7f2b9006f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:20.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.275+0000 7f2b97240700 1 -- 172.21.15.78:0/2326616350 >> 172.21.15.78:0/2326616350 conn(0x7f2b900c3a50 msgr2=0x7f2b900c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:20.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.275+0000 7f2b97240700 1 -- 172.21.15.78:0/2326616350 shutdown_connections 2023-10-14T11:59:20.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.275+0000 7f2b97240700 1 -- 172.21.15.78:0/2326616350 wait complete. 2023-10-14T11:59:20.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.276+0000 7f2b97240700 1 Processor -- start 2023-10-14T11:59:20.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.276+0000 7f2b97240700 1 -- start start 2023-10-14T11:59:20.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.277+0000 7f2b97240700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2b9006f430 0x7f2b9014fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:20.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.277+0000 7f2b97240700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b90072390 0x7f2b90150070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:20.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.277+0000 7f2b97240700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2b90155e00 0x7f2b901eb240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:20.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.277+0000 7f2b97240700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b901e9800 con 0x7f2b90072390 2023-10-14T11:59:20.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.277+0000 7f2b97240700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b90139930 con 0x7f2b9006f430 2023-10-14T11:59:20.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.277+0000 7f2b97240700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b901e8ed0 con 0x7f2b90155e00 2023-10-14T11:59:20.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.277+0000 7f2b8ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b90072390 0x7f2b90150070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:20.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.277+0000 7f2b8ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b90072390 0x7f2b90150070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46558/0 (socket says 172.21.15.78:46558) 2023-10-14T11:59:20.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.277+0000 7f2b8ffff700 1 -- 172.21.15.78:0/1522958664 learned_addr learned my addr 172.21.15.78:0/1522958664 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:20.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.277+0000 7f2b94fdc700 1 --2- 172.21.15.78:0/1522958664 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2b9006f430 0x7f2b9014fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:20.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.277+0000 7f2b957dd700 1 --2- 172.21.15.78:0/1522958664 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2b90155e00 0x7f2b901eb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:20.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.278+0000 7f2b8ffff700 1 -- 172.21.15.78:0/1522958664 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2b90155e00 msgr2=0x7f2b901eb240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:20.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.278+0000 7f2b8ffff700 1 --2- 172.21.15.78:0/1522958664 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2b90155e00 0x7f2b901eb240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:20.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.278+0000 7f2b8ffff700 1 -- 172.21.15.78:0/1522958664 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2b9006f430 msgr2=0x7f2b9014fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:20.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.278+0000 7f2b8ffff700 1 --2- 172.21.15.78:0/1522958664 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2b9006f430 0x7f2b9014fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:20.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.278+0000 7f2b8ffff700 1 -- 172.21.15.78:0/1522958664 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b88009a70 con 0x7f2b90072390 2023-10-14T11:59:20.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.278+0000 7f2b957dd700 1 --2- 172.21.15.78:0/1522958664 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2b90155e00 0x7f2b901eb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:59:20.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.278+0000 7f2b94fdc700 1 --2- 172.21.15.78:0/1522958664 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2b9006f430 0x7f2b9014fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:59:20.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.278+0000 7f2b8ffff700 1 --2- 172.21.15.78:0/1522958664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b90072390 0x7f2b90150070 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c00c540 tx=0x7f2b7c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:20.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.278+0000 7f2b8d7fa700 1 -- 172.21.15.78:0/1522958664 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c004c80 con 0x7f2b90072390 2023-10-14T11:59:20.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.278+0000 7f2b8d7fa700 1 -- 172.21.15.78:0/1522958664 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b7c004de0 con 0x7f2b90072390 2023-10-14T11:59:20.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.278+0000 7f2b97240700 1 -- 172.21.15.78:0/1522958664 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b901e91b0 con 0x7f2b90072390 2023-10-14T11:59:20.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.278+0000 7f2b8d7fa700 1 -- 172.21.15.78:0/1522958664 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c007310 con 0x7f2b90072390 2023-10-14T11:59:20.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.278+0000 7f2b97240700 1 -- 172.21.15.78:0/1522958664 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b90152470 con 0x7f2b90072390 2023-10-14T11:59:20.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.279+0000 7f2b8d7fa700 1 -- 172.21.15.78:0/1522958664 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2b7c007470 con 0x7f2b90072390 2023-10-14T11:59:20.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.280+0000 7f2b8d7fa700 1 --2- 172.21.15.78:0/1522958664 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f2b7406af00 0x7f2b7406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:20.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.280+0000 7f2b8d7fa700 1 -- 172.21.15.78:0/1522958664 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f2b7c089330 con 0x7f2b90072390 2023-10-14T11:59:20.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.280+0000 7f2b94fdc700 1 --2- 172.21.15.78:0/1522958664 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f2b7406af00 0x7f2b7406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:20.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.280+0000 7f2b94fdc700 1 --2- 172.21.15.78:0/1522958664 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f2b7406af00 0x7f2b7406d3c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2b840099f0 tx=0x7f2b84008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:20.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.282+0000 7f2ba342ab80 1 Processor -- start 2023-10-14T11:59:20.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.282+0000 7f2ba342ab80 1 -- start start 2023-10-14T11:59:20.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.282+0000 7f2ba342ab80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x564561dfd6b0 0x564561df89b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:20.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.282+0000 7f2ba342ab80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x564561dfda90 0x564561e1bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:20.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.282+0000 7f2ba342ab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564561df8ef0 0x564561e1c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:20.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.282+0000 7f2ba342ab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x564561d71c20 con 0x564561df8ef0 2023-10-14T11:59:20.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.282+0000 7f2ba342ab80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x564561d56c10 con 0x564561dfd6b0 2023-10-14T11:59:20.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.282+0000 7f2ba342ab80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x564561d54530 con 0x564561dfda90 2023-10-14T11:59:20.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.283+0000 7f2b957dd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564561df8ef0 0x564561e1c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:20.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.283+0000 7f2b957dd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564561df8ef0 0x564561e1c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46568/0 (socket says 172.21.15.78:46568) 2023-10-14T11:59:20.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.283+0000 7f2b957dd700 1 -- 172.21.15.78:0/2691356956 learned_addr learned my addr 172.21.15.78:0/2691356956 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:20.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.283+0000 7f2b8ffff700 1 --2- 172.21.15.78:0/2691356956 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x564561dfda90 0x564561e1bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:20.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.283+0000 7f2b94fdc700 1 --2- 172.21.15.78:0/2691356956 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x564561dfd6b0 0x564561df89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:20.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.283+0000 7f2b957dd700 1 -- 172.21.15.78:0/2691356956 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x564561dfda90 msgr2=0x564561e1bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:20.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.283+0000 7f2b957dd700 1 --2- 172.21.15.78:0/2691356956 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x564561dfda90 0x564561e1bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:20.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.283+0000 7f2b957dd700 1 -- 172.21.15.78:0/2691356956 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x564561dfd6b0 msgr2=0x564561df89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:20.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.283+0000 7f2b957dd700 1 --2- 172.21.15.78:0/2691356956 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x564561dfd6b0 0x564561df89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:20.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.283+0000 7f2b957dd700 1 -- 172.21.15.78:0/2691356956 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564561d0d3a0 con 0x564561df8ef0 2023-10-14T11:59:20.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.283+0000 7f2b957dd700 1 --2- 172.21.15.78:0/2691356956 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564561df8ef0 0x564561e1c1a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f2b88013000 tx=0x7f2b88010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=357b2db4fdff8f7a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:20.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.283+0000 7f2b7a7fc700 1 -- 172.21.15.78:0/2691356956 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b880070a0 con 0x564561df8ef0 2023-10-14T11:59:20.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.283+0000 7f2b7a7fc700 1 -- 172.21.15.78:0/2691356956 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b88003b20 con 0x564561df8ef0 2023-10-14T11:59:20.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.283+0000 7f2b7a7fc700 1 -- 172.21.15.78:0/2691356956 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b880125e0 con 0x564561df8ef0 2023-10-14T11:59:20.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.284+0000 7f2ba342ab80 1 -- 172.21.15.78:0/2691356956 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564561df8ef0 msgr2=0x564561e1c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:20.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.284+0000 7f2ba342ab80 1 --2- 172.21.15.78:0/2691356956 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564561df8ef0 0x564561e1c1a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f2b88013000 tx=0x7f2b88010000 comp rx=0 tx=0).stop 2023-10-14T11:59:20.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.284+0000 7f2ba342ab80 1 -- 172.21.15.78:0/2691356956 shutdown_connections 2023-10-14T11:59:20.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.284+0000 7f2ba342ab80 1 --2- 172.21.15.78:0/2691356956 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564561df8ef0 0x564561e1c1a0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:20.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.284+0000 7f2ba342ab80 1 --2- 172.21.15.78:0/2691356956 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x564561dfd6b0 0x564561df89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:20.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.284+0000 7f2ba342ab80 1 --2- 172.21.15.78:0/2691356956 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x564561dfda90 0x564561e1bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:20.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.284+0000 7f2ba342ab80 1 -- 172.21.15.78:0/2691356956 >> 172.21.15.78:0/2691356956 conn(0x564561d2ad80 msgr2=0x564561dfb3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:20.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.284+0000 7f2ba342ab80 1 -- 172.21.15.78:0/2691356956 shutdown_connections 2023-10-14T11:59:20.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.284+0000 7f2ba342ab80 1 -- 172.21.15.78:0/2691356956 wait complete. 2023-10-14T11:59:20.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.284+0000 7f2ba342ab80 1 -- start start 2023-10-14T11:59:20.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.285+0000 7f2ba342ab80 1 Processor -- start 2023-10-14T11:59:20.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.285+0000 7f2ba342ab80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x564561df8ef0 0x564561e1aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:20.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.285+0000 7f2ba342ab80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x564561dfd6b0 0x564561e3d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:20.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.285+0000 7f2ba342ab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564561dfda90 0x564561e15a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:20.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.285+0000 7f2ba342ab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x564561d4fe20 con 0x564561dfda90 2023-10-14T11:59:20.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.285+0000 7f2ba342ab80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x564561cec6c0 con 0x564561df8ef0 2023-10-14T11:59:20.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.285+0000 7f2ba342ab80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x564561cec360 con 0x564561dfd6b0 2023-10-14T11:59:20.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.285+0000 7f2ba342ab80 10 client.0 _ll_register_callbacks cb 0x564561b497b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T11:59:20.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.285+0000 7f2b957dd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564561dfda90 0x564561e15a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:20.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.285+0000 7f2b957dd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564561dfda90 0x564561e15a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46574/0 (socket says 172.21.15.78:46574) 2023-10-14T11:59:20.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.285+0000 7f2b957dd700 1 -- 172.21.15.78:0/1172835386 learned_addr learned my addr 172.21.15.78:0/1172835386 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:20.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.286+0000 7f2b8ffff700 1 --2- 172.21.15.78:0/1172835386 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x564561dfd6b0 0x564561e3d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:20.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.286+0000 7f2b94fdc700 1 --2- 172.21.15.78:0/1172835386 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x564561df8ef0 0x564561e1aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:20.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.286+0000 7f2b957dd700 1 -- 172.21.15.78:0/1172835386 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x564561dfd6b0 msgr2=0x564561e3d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:20.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.286+0000 7f2b957dd700 1 --2- 172.21.15.78:0/1172835386 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x564561dfd6b0 0x564561e3d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:20.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.286+0000 7f2b957dd700 1 -- 172.21.15.78:0/1172835386 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x564561df8ef0 msgr2=0x564561e1aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:20.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.286+0000 7f2b957dd700 1 --2- 172.21.15.78:0/1172835386 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x564561df8ef0 0x564561e1aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:20.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.286+0000 7f2b957dd700 1 -- 172.21.15.78:0/1172835386 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b90152df0 con 0x564561dfda90 2023-10-14T11:59:20.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.286+0000 7f2b957dd700 1 --2- 172.21.15.78:0/1172835386 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564561dfda90 0x564561e15a30 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f2b88009fd0 tx=0x7f2b88004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3228157f02d36f8 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:20.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.286+0000 7f2b6f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-14T11:59:20.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.286+0000 7f2b6f7fe700 1 -- 172.21.15.78:0/1172835386 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b88004600 con 0x564561dfda90 2023-10-14T11:59:20.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.286+0000 7f2b6f7fe700 1 -- 172.21.15.78:0/1172835386 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f2b88004760 con 0x564561dfda90 2023-10-14T11:59:20.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.286+0000 7f2ba342ab80 1 -- 172.21.15.78:0/1172835386 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564561df05b0 con 0x564561dfda90 2023-10-14T11:59:20.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.286+0000 7f2b6f7fe700 1 -- 172.21.15.78:0/1172835386 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2b88004920 con 0x564561dfda90 2023-10-14T11:59:20.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.286+0000 7f2ba342ab80 10 client.5292 fetch_fsmap learned FSMap version 17 2023-10-14T11:59:20.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.286+0000 7f2ba342ab80 1 -- 172.21.15.78:0/1172835386 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x564561b7ad90 con 0x564561dfda90 2023-10-14T11:59:20.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.286+0000 7f2b6f7fe700 1 -- 172.21.15.78:0/1172835386 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x564561df05b0 con 0x564561dfda90 2023-10-14T11:59:20.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.286+0000 7f2b6f7fe700 1 -- 172.21.15.78:0/1172835386 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 17) v1 ==== 3299+0+0 (secure 0 0 0) 0x7f2b880124b0 con 0x564561dfda90 2023-10-14T11:59:20.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.286+0000 7f2ba342ab80 10 client.5292 fetch_fsmap finished waiting for FSMap version 17 2023-10-14T11:59:20.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.286+0000 7f2ba342ab80 10 client.5292 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-14T11:59:20.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.286+0000 7f2ba342ab80 20 client.5292 populate_metadata read hostname 'smithi078' 2023-10-14T11:59:20.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.286+0000 7f2ba342ab80 1 --2- 172.21.15.78:0/1172835386 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x564561e15f70 0x564561e17ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:20.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.286+0000 7f2ba342ab80 4 client.5292 mds_command: new command op to 5031 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-14T11:59:20.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.286+0000 7f2ba342ab80 1 -- 172.21.15.78:0/1172835386 --> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564561d48e00 con 0x564561e15f70 2023-10-14T11:59:20.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.287+0000 7f2b94fdc700 1 --2- 172.21.15.78:0/1172835386 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x564561e15f70 0x564561e17ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:20.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.287+0000 7f2b94fdc700 1 --2- 172.21.15.78:0/1172835386 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x564561e15f70 0x564561e17ba0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4a2b5b47a07b0bf server_cookie=dd11cfe5048bba72 in_seq=0 out_seq=0 2023-10-14T11:59:20.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.287+0000 7f2b6f7fe700 10 client.5292 ms_handle_connect on v2:172.21.15.195:6833/2130671508 2023-10-14T11:59:20.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.288+0000 7f2b6f7fe700 1 -- 172.21.15.78:0/1172835386 <== mds.0 v2:172.21.15.195:6833/2130671508 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x564561d48e00 con 0x564561e15f70 2023-10-14T11:59:20.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.288+0000 7f2b6f7fe700 10 client.5292 handle_command_reply: tid=0 2023-10-14T11:59:20.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.288+0000 7f2ba342ab80 1 client.5292 shutdown 2023-10-14T11:59:20.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.288+0000 7f2ba342ab80 1 -- 172.21.15.78:0/1172835386 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564561dfda90 msgr2=0x564561e15a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:20.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.288+0000 7f2ba342ab80 1 --2- 172.21.15.78:0/1172835386 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564561dfda90 0x564561e15a30 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f2b88009fd0 tx=0x7f2b88004b40 comp rx=0 tx=0).stop 2023-10-14T11:59:20.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.288+0000 7f2ba342ab80 1 -- 172.21.15.78:0/1172835386 shutdown_connections 2023-10-14T11:59:20.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.288+0000 7f2ba342ab80 1 --2- 172.21.15.78:0/1172835386 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564561dfda90 0x564561e15a30 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:20.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.288+0000 7f2ba342ab80 1 --2- 172.21.15.78:0/1172835386 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x564561e15f70 0x564561e17ba0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:20.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.288+0000 7f2ba342ab80 1 --2- 172.21.15.78:0/1172835386 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x564561df8ef0 0x564561e1aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:20.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.288+0000 7f2b6f7fe700 0 client.5292 ms_handle_reset on v2:172.21.15.195:6833/2130671508 2023-10-14T11:59:20.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.288+0000 7f2ba342ab80 1 --2- 172.21.15.78:0/1172835386 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x564561dfd6b0 0x564561e3d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:20.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.288+0000 7f2ba342ab80 1 -- 172.21.15.78:0/1172835386 >> 172.21.15.78:0/1172835386 conn(0x564561d2ad80 msgr2=0x564561e1ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:20.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.288+0000 7f2ba342ab80 1 -- 172.21.15.78:0/1172835386 shutdown_connections 2023-10-14T11:59:20.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.288+0000 7f2ba342ab80 1 -- 172.21.15.78:0/1172835386 wait complete. 2023-10-14T11:59:20.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.288+0000 7f2ba342ab80 20 client.5292 trim_cache size 0 max 16384 2023-10-14T11:59:20.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.306+0000 7f2ba342ab80 1 Processor -- start 2023-10-14T11:59:20.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.306+0000 7f2ba342ab80 1 -- start start 2023-10-14T11:59:20.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.306+0000 7f2ba342ab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564561df8ef0 0x564561e1fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:20.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.306+0000 7f2ba342ab80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x564561dfd6b0 0x564561df1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:20.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.306+0000 7f2ba342ab80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x564561dfda90 0x564561df2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:20.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.306+0000 7f2ba342ab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x564561bd08d0 con 0x564561df8ef0 2023-10-14T11:59:20.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.306+0000 7f2ba342ab80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x564561df29c0 con 0x564561dfda90 2023-10-14T11:59:20.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.306+0000 7f2ba342ab80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x564561e3d7e0 con 0x564561dfd6b0 2023-10-14T11:59:20.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.306+0000 7f2b94fdc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564561df8ef0 0x564561e1fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:20.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.306+0000 7f2b94fdc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564561df8ef0 0x564561e1fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46578/0 (socket says 172.21.15.78:46578) 2023-10-14T11:59:20.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.306+0000 7f2b94fdc700 1 -- 172.21.15.78:0/4122778332 learned_addr learned my addr 172.21.15.78:0/4122778332 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:20.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.307+0000 7f2b94fdc700 1 -- 172.21.15.78:0/4122778332 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x564561dfd6b0 msgr2=0x564561df1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:20.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.307+0000 7f2b94fdc700 1 --2- 172.21.15.78:0/4122778332 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x564561dfd6b0 0x564561df1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:20.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.307+0000 7f2b94fdc700 1 -- 172.21.15.78:0/4122778332 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x564561dfda90 msgr2=0x564561df2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:20.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.307+0000 7f2b94fdc700 1 --2- 172.21.15.78:0/4122778332 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x564561dfda90 0x564561df2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:20.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.307+0000 7f2b94fdc700 1 -- 172.21.15.78:0/4122778332 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b9006f2a0 con 0x564561df8ef0 2023-10-14T11:59:20.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.307+0000 7f2b94fdc700 1 --2- 172.21.15.78:0/4122778332 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564561df8ef0 0x564561e1fb50 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f2b840094b0 tx=0x7f2b84016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a348e8d30239faac server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:20.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.307+0000 7f2b7affd700 1 -- 172.21.15.78:0/4122778332 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8402b510 con 0x564561df8ef0 2023-10-14T11:59:20.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.307+0000 7f2b7affd700 1 -- 172.21.15.78:0/4122778332 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b8402a040 con 0x564561df8ef0 2023-10-14T11:59:20.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.307+0000 7f2b7affd700 1 -- 172.21.15.78:0/4122778332 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8402ba10 con 0x564561df8ef0 2023-10-14T11:59:20.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.307+0000 7f2ba342ab80 1 -- 172.21.15.78:0/4122778332 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564561df8ef0 msgr2=0x564561e1fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:20.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.307+0000 7f2ba342ab80 1 --2- 172.21.15.78:0/4122778332 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564561df8ef0 0x564561e1fb50 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f2b840094b0 tx=0x7f2b84016000 comp rx=0 tx=0).stop 2023-10-14T11:59:20.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.307+0000 7f2ba342ab80 1 -- 172.21.15.78:0/4122778332 shutdown_connections 2023-10-14T11:59:20.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.307+0000 7f2ba342ab80 1 --2- 172.21.15.78:0/4122778332 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x564561dfda90 0x564561df2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:20.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.307+0000 7f2ba342ab80 1 --2- 172.21.15.78:0/4122778332 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564561df8ef0 0x564561e1fb50 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:20.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.308+0000 7f2ba342ab80 1 --2- 172.21.15.78:0/4122778332 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x564561dfd6b0 0x564561df1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:20.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.308+0000 7f2ba342ab80 1 -- 172.21.15.78:0/4122778332 >> 172.21.15.78:0/4122778332 conn(0x564561d2ad80 msgr2=0x564561e601d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:20.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.308+0000 7f2ba342ab80 1 -- 172.21.15.78:0/4122778332 shutdown_connections 2023-10-14T11:59:20.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.308+0000 7f2ba342ab80 1 -- 172.21.15.78:0/4122778332 wait complete. 2023-10-14T11:59:20.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.308+0000 7f2ba342ab80 1 -- start start 2023-10-14T11:59:20.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.308+0000 7f2ba342ab80 1 Processor -- start 2023-10-14T11:59:20.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.309+0000 7f2ba342ab80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x564561df8ef0 0x564561e4ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:20.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.309+0000 7f2ba342ab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564561dfd6b0 0x564561ea8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:20.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.309+0000 7f2ba342ab80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x564561dfda90 0x564561ea2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:20.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.309+0000 7f2ba342ab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x564561ea9280 con 0x564561dfd6b0 2023-10-14T11:59:20.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.309+0000 7f2ba342ab80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x564561e3d0a0 con 0x564561df8ef0 2023-10-14T11:59:20.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.309+0000 7f2ba342ab80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x564561dfc8a0 con 0x564561dfda90 2023-10-14T11:59:20.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.310+0000 7f2ba342ab80 10 client.0 _ll_register_callbacks cb 0x7f2b84009720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T11:59:20.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.310+0000 7f2b8ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564561dfd6b0 0x564561ea8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:20.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.310+0000 7f2b8ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564561dfd6b0 0x564561ea8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46586/0 (socket says 172.21.15.78:46586) 2023-10-14T11:59:20.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.310+0000 7f2b8ffff700 1 -- 172.21.15.78:0/3961504090 learned_addr learned my addr 172.21.15.78:0/3961504090 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:20.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.310+0000 7f2b957dd700 1 --2- 172.21.15.78:0/3961504090 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x564561dfda90 0x564561ea2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:20.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.310+0000 7f2b94fdc700 1 --2- 172.21.15.78:0/3961504090 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x564561df8ef0 0x564561e4ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:20.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.310+0000 7f2b8ffff700 1 -- 172.21.15.78:0/3961504090 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x564561dfda90 msgr2=0x564561ea2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:20.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.310+0000 7f2b8ffff700 1 --2- 172.21.15.78:0/3961504090 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x564561dfda90 0x564561ea2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:20.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.310+0000 7f2b8ffff700 1 -- 172.21.15.78:0/3961504090 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x564561df8ef0 msgr2=0x564561e4ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:20.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.310+0000 7f2b8ffff700 1 --2- 172.21.15.78:0/3961504090 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x564561df8ef0 0x564561e4ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:20.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.310+0000 7f2b8ffff700 1 -- 172.21.15.78:0/3961504090 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b8400e050 con 0x564561dfd6b0 2023-10-14T11:59:20.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.310+0000 7f2b94fdc700 1 --2- 172.21.15.78:0/3961504090 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x564561df8ef0 0x564561e4ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:59:20.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.310+0000 7f2b957dd700 1 --2- 172.21.15.78:0/3961504090 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x564561dfda90 0x564561ea2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:59:20.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.310+0000 7f2b8ffff700 1 --2- 172.21.15.78:0/3961504090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564561dfd6b0 0x564561ea8800 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c054610 tx=0x7f2b7c0077a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa683d4639809cdf server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:20.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.310+0000 7f2b6f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-14T11:59:20.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.310+0000 7f2b6f7fe700 1 -- 172.21.15.78:0/3961504090 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c05c260 con 0x564561dfd6b0 2023-10-14T11:59:20.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.310+0000 7f2b6f7fe700 1 -- 172.21.15.78:0/3961504090 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f2b7c060410 con 0x564561dfd6b0 2023-10-14T11:59:20.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.310+0000 7f2ba342ab80 1 -- 172.21.15.78:0/3961504090 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2b88003b20 con 0x564561dfd6b0 2023-10-14T11:59:20.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.310+0000 7f2b6f7fe700 1 -- 172.21.15.78:0/3961504090 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2b7c0890b0 con 0x564561dfd6b0 2023-10-14T11:59:20.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.310+0000 7f2ba342ab80 10 client.5298 fetch_fsmap learned FSMap version 17 2023-10-14T11:59:20.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.310+0000 7f2ba342ab80 1 -- 172.21.15.78:0/3961504090 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x564561b63c10 con 0x564561dfd6b0 2023-10-14T11:59:20.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.310+0000 7f2b6f7fe700 1 -- 172.21.15.78:0/3961504090 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b7c07a030 con 0x564561dfd6b0 2023-10-14T11:59:20.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.310+0000 7f2b6f7fe700 1 -- 172.21.15.78:0/3961504090 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 17) v1 ==== 3299+0+0 (secure 0 0 0) 0x7f2b7c058370 con 0x564561dfd6b0 2023-10-14T11:59:20.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.310+0000 7f2ba342ab80 10 client.5298 fetch_fsmap finished waiting for FSMap version 17 2023-10-14T11:59:20.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.310+0000 7f2ba342ab80 10 client.5298 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-14T11:59:20.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.310+0000 7f2ba342ab80 20 client.5298 populate_metadata read hostname 'smithi078' 2023-10-14T11:59:20.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.310+0000 7f2ba342ab80 1 --2- 172.21.15.78:0/3961504090 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x564561ea2b60 0x564561e16350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:20.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.311+0000 7f2ba342ab80 4 client.5298 mds_command: new command op to 5031 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-14T11:59:20.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.311+0000 7f2ba342ab80 1 -- 172.21.15.78:0/3961504090 --> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] -- command(tid 0: {"prefix": "status"}) v1 -- 0x564561d48b20 con 0x564561ea2b60 2023-10-14T11:59:20.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.311+0000 7f2b94fdc700 1 --2- 172.21.15.78:0/3961504090 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x564561ea2b60 0x564561e16350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:20.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.311+0000 7f2b94fdc700 1 --2- 172.21.15.78:0/3961504090 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x564561ea2b60 0x564561e16350 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b24fcf97624f4c63 server_cookie=e31b0b417c978086 in_seq=0 out_seq=0 2023-10-14T11:59:20.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.311+0000 7f2b6f7fe700 10 client.5298 ms_handle_connect on v2:172.21.15.195:6833/2130671508 2023-10-14T11:59:20.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.311+0000 7f2b6f7fe700 1 -- 172.21.15.78:0/3961504090 <== mds.0 v2:172.21.15.195:6833/2130671508 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x564561d48b20 con 0x564561ea2b60 2023-10-14T11:59:20.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.311+0000 7f2b6f7fe700 10 client.5298 handle_command_reply: tid=0 2023-10-14T11:59:20.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.311+0000 7f2ba342ab80 1 client.5298 shutdown 2023-10-14T11:59:20.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.311+0000 7f2ba342ab80 1 -- 172.21.15.78:0/3961504090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564561dfd6b0 msgr2=0x564561ea8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:20.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.311+0000 7f2ba342ab80 1 --2- 172.21.15.78:0/3961504090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564561dfd6b0 0x564561ea8800 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c054610 tx=0x7f2b7c0077a0 comp rx=0 tx=0).stop 2023-10-14T11:59:20.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.311+0000 7f2ba342ab80 1 -- 172.21.15.78:0/3961504090 shutdown_connections 2023-10-14T11:59:20.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.311+0000 7f2ba342ab80 1 --2- 172.21.15.78:0/3961504090 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x564561dfda90 0x564561ea2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:20.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.311+0000 7f2ba342ab80 1 --2- 172.21.15.78:0/3961504090 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x564561ea2b60 0x564561e16350 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:20.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.311+0000 7f2ba342ab80 1 --2- 172.21.15.78:0/3961504090 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x564561df8ef0 0x564561e4ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:20.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.311+0000 7f2ba342ab80 1 --2- 172.21.15.78:0/3961504090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564561dfd6b0 0x564561ea8800 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:20.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.311+0000 7f2b6f7fe700 0 client.5298 ms_handle_reset on v2:172.21.15.195:6833/2130671508 2023-10-14T11:59:20.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.311+0000 7f2ba342ab80 1 -- 172.21.15.78:0/3961504090 >> 172.21.15.78:0/3961504090 conn(0x564561d2ad80 msgr2=0x564561df1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:20.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.311+0000 7f2ba342ab80 1 -- 172.21.15.78:0/3961504090 shutdown_connections 2023-10-14T11:59:20.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.312+0000 7f2ba342ab80 1 -- 172.21.15.78:0/3961504090 wait complete. 2023-10-14T11:59:20.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.312+0000 7f2ba342ab80 20 client.5298 trim_cache size 0 max 16384 2023-10-14T11:59:20.341 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-14T11:59:20.341 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "4c5dc134-3092-406a-a1e1-13e6d9e926e8", 2023-10-14T11:59:20.341 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2023-10-14T11:59:20.342 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 5031, 2023-10-14T11:59:20.342 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2023-10-14T11:59:20.342 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2023-10-14T11:59:20.342 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "testcephfs1", 2023-10-14T11:59:20.342 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 18.127444049000001, 2023-10-14T11:59:20.343 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 17, 2023-10-14T11:59:20.343 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 32, 2023-10-14T11:59:20.343 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 32, 2023-10-14T11:59:20.343 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 26.018266165 2023-10-14T11:59:20.343 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-14T11:59:20.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.312+0000 7f2b97240700 1 -- 172.21.15.78:0/1522958664 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f2b7406af00 msgr2=0x7f2b7406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:20.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.312+0000 7f2b97240700 1 --2- 172.21.15.78:0/1522958664 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f2b7406af00 0x7f2b7406d3c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2b840099f0 tx=0x7f2b84008040 comp rx=0 tx=0).stop 2023-10-14T11:59:20.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.312+0000 7f2b97240700 1 -- 172.21.15.78:0/1522958664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b90072390 msgr2=0x7f2b90150070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:20.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.312+0000 7f2b97240700 1 --2- 172.21.15.78:0/1522958664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b90072390 0x7f2b90150070 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c00c540 tx=0x7f2b7c00c900 comp rx=0 tx=0).stop 2023-10-14T11:59:20.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.312+0000 7f2b97240700 1 -- 172.21.15.78:0/1522958664 shutdown_connections 2023-10-14T11:59:20.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.312+0000 7f2b97240700 1 --2- 172.21.15.78:0/1522958664 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2b90155e00 0x7f2b901eb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:20.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.312+0000 7f2b97240700 1 --2- 172.21.15.78:0/1522958664 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f2b7406af00 0x7f2b7406d3c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:20.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.312+0000 7f2b97240700 1 --2- 172.21.15.78:0/1522958664 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2b9006f430 0x7f2b9014fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:20.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.312+0000 7f2b97240700 1 --2- 172.21.15.78:0/1522958664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b90072390 0x7f2b90150070 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:20.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.312+0000 7f2b97240700 1 -- 172.21.15.78:0/1522958664 >> 172.21.15.78:0/1522958664 conn(0x7f2b900c3a50 msgr2=0x7f2b900c2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:20.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.312+0000 7f2b97240700 1 -- 172.21.15.78:0/1522958664 shutdown_connections 2023-10-14T11:59:20.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:20.312+0000 7f2b97240700 1 -- 172.21.15.78:0/1522958664 wait complete. 2023-10-14T11:59:20.349 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 17, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-14T11:59:01.562764+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 17, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "testcephfs1", "in": [ 0 ], "info": { "gid_5031": { "addr": "172.21.15.195:6835/2130671508", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6833", "nonce": 2130671508, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2130671508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5031, "incarnation": 13, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2023-10-14T11:59:04.592935+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5031 } } } ], "standbys": [ { "addr": "172.21.15.195:6834/1492486786", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6832", "nonce": 1492486786, "type": "v2" }, { "addr": "172.21.15.195:6834", "nonce": 1492486786, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4658, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6836/2991384297", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6834", "nonce": 2991384297, "type": "v2" }, { "addr": "172.21.15.78:6836", "nonce": 2991384297, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5016, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6837/3246640037", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6835", "nonce": 3246640037, "type": "v2" }, { "addr": "172.21.15.78:6837", "nonce": 3246640037, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5019, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-14T11:59:20.350 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 2, 'name': 'b', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2023-10-14T11:59:20.351 INFO:tasks.cephfs.test_failover:current = [{'gid': 5019, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6837/3246640037', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 3246640037}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 3246640037}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 4658, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.195:6834/1492486786', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6832', 'nonce': 1492486786}, {'type': 'v1', 'addr': '172.21.15.195:6834', 'nonce': 1492486786}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 5016, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6836/2991384297', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 2991384297}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 2991384297}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 5031, 'name': 'd', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.195:6835/2130671508', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6833', 'nonce': 2130671508}, {'type': 'v1', 'addr': '172.21.15.195:6835', 'nonce': 2130671508}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2023-10-14T11:59:20.351 DEBUG:tasks.cephfs.test_failover:2 != -1 2023-10-14T11:59:20.352 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2023-10-14T11:59:23.618 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T11:59:23.621 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T11:59:25.353 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T11:59:25.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.488+0000 7f606924f700 1 Processor -- start 2023-10-14T11:59:25.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.488+0000 7f606924f700 1 -- start start 2023-10-14T11:59:25.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.488+0000 7f606924f700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f60640c44c0 0x7f60640c7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:25.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.488+0000 7f606924f700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f60640c72a0 0x7f60640c86e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:25.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.488+0000 7f606924f700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6064156b20 con 0x7f60640c8c20 2023-10-14T11:59:25.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.488+0000 7f606924f700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6064155f10 con 0x7f60640c72a0 2023-10-14T11:59:25.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.488+0000 7f606924f700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60641561f0 con 0x7f60640c44c0 2023-10-14T11:59:25.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.488+0000 7f606259c700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f60640c72a0 0x7f60640c86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:25.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.489+0000 7f606259c700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f60640c72a0 0x7f60640c86e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46594/0 (socket says 172.21.15.78:46594) 2023-10-14T11:59:25.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.489+0000 7f606259c700 1 -- 172.21.15.78:0/1135632560 learned_addr learned my addr 172.21.15.78:0/1135632560 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:25.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.489+0000 7f6062d9d700 1 --2- 172.21.15.78:0/1135632560 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f60640c44c0 0x7f60640c7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:25.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.489+0000 7f606259c700 1 -- 172.21.15.78:0/1135632560 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f60640c44c0 msgr2=0x7f60640c7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:25.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.489+0000 7f606259c700 1 --2- 172.21.15.78:0/1135632560 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f60640c44c0 0x7f60640c7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:25.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.489+0000 7f606259c700 1 -- 172.21.15.78:0/1135632560 >> v1:172.21.15.195:6789/0 conn(0x7f60640c8c20 legacy=0x7f60640c2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:25.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.489+0000 7f606259c700 1 -- 172.21.15.78:0/1135632560 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60641564d0 con 0x7f60640c72a0 2023-10-14T11:59:25.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.489+0000 7f6062d9d700 1 --2- 172.21.15.78:0/1135632560 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f60640c44c0 0x7f60640c7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:59:25.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.489+0000 7f606259c700 1 --2- 172.21.15.78:0/1135632560 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60640c72a0 0x7f60640c86e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f60580098f0 tx=0x7f6058009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35830b018683ad1f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:25.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.489+0000 7f6061d9b700 1 -- 172.21.15.78:0/1135632560 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f605800b8a0 con 0x7f60640c72a0 2023-10-14T11:59:25.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.489+0000 7f6061d9b700 1 -- 172.21.15.78:0/1135632560 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f605800ba00 con 0x7f60640c72a0 2023-10-14T11:59:25.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.489+0000 7f6061d9b700 1 -- 172.21.15.78:0/1135632560 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f605800bd90 con 0x7f60640c72a0 2023-10-14T11:59:25.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.490+0000 7f606924f700 1 -- 172.21.15.78:0/1135632560 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60640c72a0 msgr2=0x7f60640c86e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:25.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.490+0000 7f606924f700 1 --2- 172.21.15.78:0/1135632560 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60640c72a0 0x7f60640c86e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f60580098f0 tx=0x7f6058009c00 comp rx=0 tx=0).stop 2023-10-14T11:59:25.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.490+0000 7f606924f700 1 -- 172.21.15.78:0/1135632560 shutdown_connections 2023-10-14T11:59:25.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.490+0000 7f606924f700 1 --2- 172.21.15.78:0/1135632560 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f60640c44c0 0x7f60640c7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:25.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.490+0000 7f606924f700 1 --2- 172.21.15.78:0/1135632560 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60640c72a0 0x7f60640c86e0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:25.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.490+0000 7f606924f700 1 -- 172.21.15.78:0/1135632560 >> 172.21.15.78:0/1135632560 conn(0x7f60640737d0 msgr2=0x7f6064073be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:25.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.490+0000 7f606924f700 1 -- 172.21.15.78:0/1135632560 shutdown_connections 2023-10-14T11:59:25.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.490+0000 7f606924f700 1 -- 172.21.15.78:0/1135632560 wait complete. 2023-10-14T11:59:25.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.491+0000 7f606924f700 1 Processor -- start 2023-10-14T11:59:25.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.491+0000 7f606924f700 1 -- start start 2023-10-14T11:59:25.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.492+0000 7f606924f700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f60640c44c0 0x7f60641ea340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:25.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.492+0000 7f606924f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60640c72a0 0x7f60640c15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:25.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.492+0000 7f606924f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60640c8c20 0x7f60640be1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:25.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.492+0000 7f606924f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60641ecf30 con 0x7f60640c8c20 2023-10-14T11:59:25.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.492+0000 7f606924f700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6064150ac0 con 0x7f60640c72a0 2023-10-14T11:59:25.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.492+0000 7f606924f700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60641ea880 con 0x7f60640c44c0 2023-10-14T11:59:25.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.492+0000 7f606359e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60640c8c20 0x7f60640be1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:25.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.492+0000 7f606359e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60640c8c20 0x7f60640be1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46602/0 (socket says 172.21.15.78:46602) 2023-10-14T11:59:25.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.492+0000 7f606359e700 1 -- 172.21.15.78:0/1799413457 learned_addr learned my addr 172.21.15.78:0/1799413457 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:25.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.492+0000 7f6062d9d700 1 --2- 172.21.15.78:0/1799413457 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f60640c44c0 0x7f60641ea340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:25.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.492+0000 7f606259c700 1 --2- 172.21.15.78:0/1799413457 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60640c72a0 0x7f60640c15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:25.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.492+0000 7f606359e700 1 -- 172.21.15.78:0/1799413457 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f60640c44c0 msgr2=0x7f60641ea340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:25.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.492+0000 7f606359e700 1 --2- 172.21.15.78:0/1799413457 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f60640c44c0 0x7f60641ea340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:25.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.492+0000 7f606359e700 1 -- 172.21.15.78:0/1799413457 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60640c72a0 msgr2=0x7f60640c15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:25.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.492+0000 7f606359e700 1 --2- 172.21.15.78:0/1799413457 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60640c72a0 0x7f60640c15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:25.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.492+0000 7f606359e700 1 -- 172.21.15.78:0/1799413457 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6058009550 con 0x7f60640c8c20 2023-10-14T11:59:25.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.493+0000 7f606359e700 1 --2- 172.21.15.78:0/1799413457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60640c8c20 0x7f60640be1d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f605c00cb50 tx=0x7f605c00ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:25.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.493+0000 7f604b7fe700 1 -- 172.21.15.78:0/1799413457 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f605c003c00 con 0x7f60640c8c20 2023-10-14T11:59:25.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.493+0000 7f604b7fe700 1 -- 172.21.15.78:0/1799413457 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f605c003d60 con 0x7f60640c8c20 2023-10-14T11:59:25.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.493+0000 7f604b7fe700 1 -- 172.21.15.78:0/1799413457 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f605c00aa20 con 0x7f60640c8c20 2023-10-14T11:59:25.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.493+0000 7f606924f700 1 -- 172.21.15.78:0/1799413457 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f606406d070 con 0x7f60640c8c20 2023-10-14T11:59:25.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.493+0000 7f606924f700 1 -- 172.21.15.78:0/1799413457 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f606406d630 con 0x7f60640c8c20 2023-10-14T11:59:25.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.494+0000 7f606924f700 1 -- 172.21.15.78:0/1799413457 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60641ed0a0 con 0x7f60640c8c20 2023-10-14T11:59:25.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.494+0000 7f604b7fe700 1 -- 172.21.15.78:0/1799413457 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f605c00ab80 con 0x7f60640c8c20 2023-10-14T11:59:25.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.495+0000 7f604b7fe700 1 --2- 172.21.15.78:0/1799413457 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f605006af00 0x7f605006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:25.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.495+0000 7f6062d9d700 1 --2- 172.21.15.78:0/1799413457 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f605006af00 0x7f605006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:25.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.495+0000 7f604b7fe700 1 -- 172.21.15.78:0/1799413457 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f605c089560 con 0x7f60640c8c20 2023-10-14T11:59:25.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.495+0000 7f6062d9d700 1 --2- 172.21.15.78:0/1799413457 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f605006af00 0x7f605006d3c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f60540099d0 tx=0x7f6054008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:25.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.499+0000 7f604b7fe700 1 -- 172.21.15.78:0/1799413457 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f605c054680 con 0x7f60640c8c20 2023-10-14T11:59:25.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.733+0000 7f606924f700 1 -- 172.21.15.78:0/1799413457 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6064074b80 con 0x7f60640c8c20 2023-10-14T11:59:25.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.733+0000 7f604b7fe700 1 -- 172.21.15.78:0/1799413457 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4924 (secure 0 0 0) 0x7f605c057ca0 con 0x7f60640c8c20 2023-10-14T11:59:25.736 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:59:25.736 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4658,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6834/1492486786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1492486786},{"type":"v1","addr":"172.21.15.195:6834","nonce":1492486786}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5016,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/2991384297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2991384297},{"type":"v1","addr":"172.21.15.78:6836","nonce":2991384297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5019,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3246640037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3246640037},{"type":"v1","addr":"172.21.15.78:6837","nonce":3246640037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T11:59:01.562764+0000","modified":"2023-10-14T11:59:04.592935+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5031":{"gid":5031,"name":"d","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.195:6835/2130671508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6833","nonce":2130671508},{"type":"v1","addr":"172.21.15.195:6835","nonce":2130671508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"testcephfs1","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-14T11:59:25.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.735+0000 7f606924f700 1 -- 172.21.15.78:0/1799413457 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f605006af00 msgr2=0x7f605006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:25.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.735+0000 7f606924f700 1 --2- 172.21.15.78:0/1799413457 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f605006af00 0x7f605006d3c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f60540099d0 tx=0x7f6054008040 comp rx=0 tx=0).stop 2023-10-14T11:59:25.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.735+0000 7f606924f700 1 -- 172.21.15.78:0/1799413457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60640c8c20 msgr2=0x7f60640be1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:25.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.735+0000 7f606924f700 1 --2- 172.21.15.78:0/1799413457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60640c8c20 0x7f60640be1d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f605c00cb50 tx=0x7f605c00ce60 comp rx=0 tx=0).stop 2023-10-14T11:59:25.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.736+0000 7f606924f700 1 -- 172.21.15.78:0/1799413457 shutdown_connections 2023-10-14T11:59:25.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.736+0000 7f606924f700 1 --2- 172.21.15.78:0/1799413457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60640c8c20 0x7f60640be1d0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:25.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.736+0000 7f606924f700 1 --2- 172.21.15.78:0/1799413457 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f60640c44c0 0x7f60641ea340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:25.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.736+0000 7f606924f700 1 --2- 172.21.15.78:0/1799413457 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f605006af00 0x7f605006d3c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:25.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.736+0000 7f606924f700 1 --2- 172.21.15.78:0/1799413457 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60640c72a0 0x7f60640c15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:25.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.736+0000 7f606924f700 1 -- 172.21.15.78:0/1799413457 >> 172.21.15.78:0/1799413457 conn(0x7f60640737d0 msgr2=0x7f60640c6520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:25.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.736+0000 7f606924f700 1 -- 172.21.15.78:0/1799413457 shutdown_connections 2023-10-14T11:59:25.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:25.736+0000 7f606924f700 1 -- 172.21.15.78:0/1799413457 wait complete. 2023-10-14T11:59:25.740 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 17 2023-10-14T11:59:26.152 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-14T11:59:01.562764+0000', 'modified': '2023-10-14T11:59:04.592935+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5031}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5031': {'gid': 5031, 'name': 'd', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.195:6835/2130671508', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6833', 'nonce': 2130671508}, {'type': 'v1', 'addr': '172.21.15.195:6835', 'nonce': 2130671508}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'testcephfs1', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-14T11:59:26.152 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-14T11:59:26.153 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-14T11:59:26.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.275+0000 7f61eb17b700 1 Processor -- start 2023-10-14T11:59:26.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.275+0000 7f61eb17b700 1 -- start start 2023-10-14T11:59:26.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.276+0000 7f61eb17b700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f61e40c1b70 0x7f61e40a9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:26.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.276+0000 7f61eb17b700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61e406e770 con 0x7f61e406dc80 2023-10-14T11:59:26.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.276+0000 7f61eb17b700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61e40a9780 con 0x7f61e40aa150 2023-10-14T11:59:26.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.276+0000 7f61eb17b700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61e4071b60 con 0x7f61e40c1b70 2023-10-14T11:59:26.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.276+0000 7f61e8f17700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f61e406dc80 0x7f61e406e060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51482/0 (socket says 172.21.15.78:51482) 2023-10-14T11:59:26.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.276+0000 7f61e8f17700 1 -- 172.21.15.78:0/17353623 learned_addr learned my addr 172.21.15.78:0/17353623 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:59:26.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.276+0000 7f61e3fff700 1 --2- 172.21.15.78:0/17353623 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f61e40c1b70 0x7f61e40a9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:26.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.276+0000 7f61e37fe700 1 -- 172.21.15.78:0/17353623 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2948098060 0 0) 0x7f61e406e770 con 0x7f61e406dc80 2023-10-14T11:59:26.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.276+0000 7f61e37fe700 1 -- 172.21.15.78:0/17353623 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61d40035b0 con 0x7f61e406dc80 2023-10-14T11:59:26.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.276+0000 7f61e37fe700 1 -- 172.21.15.78:0/17353623 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3443909748 0 0) 0x7f61d40035b0 con 0x7f61e406dc80 2023-10-14T11:59:26.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.276+0000 7f61e37fe700 1 -- 172.21.15.78:0/17353623 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f61e40c1b70 msgr2=0x7f61e40a9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:26.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.276+0000 7f61e37fe700 1 --2- 172.21.15.78:0/17353623 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f61e40c1b70 0x7f61e40a9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:26.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.276+0000 7f61e37fe700 1 -- 172.21.15.78:0/17353623 >> v1:172.21.15.195:6789/0 conn(0x7f61e40aa150 legacy=0x7f61e40c2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:26.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.276+0000 7f61e37fe700 1 -- 172.21.15.78:0/17353623 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e40c1340 con 0x7f61e406dc80 2023-10-14T11:59:26.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.276+0000 7f61e37fe700 1 -- 172.21.15.78:0/17353623 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441799917 0 0) 0x7f61d8001670 con 0x7f61e406dc80 2023-10-14T11:59:26.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.276+0000 7f61e37fe700 1 -- 172.21.15.78:0/17353623 >> v1:172.21.15.78:6789/0 conn(0x7f61e406dc80 legacy=0x7f61e406e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:26.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.277+0000 7f61e37fe700 1 --2- 172.21.15.78:0/17353623 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f61e40c1b70 0x7f61d4003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:26.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.277+0000 7f61e37fe700 1 --2- 172.21.15.78:0/17353623 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61d4003fb0 0x7f61d4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:26.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.277+0000 7f61e37fe700 1 --2- 172.21.15.78:0/17353623 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f61d4004d90 0x7f61d4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:26.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.277+0000 7f61e37fe700 1 -- 172.21.15.78:0/17353623 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e4071b60 con 0x7f61d4003fb0 2023-10-14T11:59:26.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.277+0000 7f61e37fe700 1 -- 172.21.15.78:0/17353623 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d400baf0 con 0x7f61d4004d90 2023-10-14T11:59:26.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.277+0000 7f61e37fe700 1 -- 172.21.15.78:0/17353623 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61d400bcb0 con 0x7f61e40c1b70 2023-10-14T11:59:26.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.277+0000 7f61e3fff700 1 --2- 172.21.15.78:0/17353623 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61d4003fb0 0x7f61d4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:26.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.277+0000 7f61e8f17700 1 --2- 172.21.15.78:0/17353623 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f61e40c1b70 0x7f61d4003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:26.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.277+0000 7f61e9718700 1 --2- 172.21.15.78:0/17353623 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f61d4004d90 0x7f61d4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:26.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.277+0000 7f61e3fff700 1 -- 172.21.15.78:0/17353623 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f61e40c1b70 msgr2=0x7f61d4003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:26.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.277+0000 7f61e3fff700 1 --2- 172.21.15.78:0/17353623 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f61e40c1b70 0x7f61d4003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:26.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.277+0000 7f61e3fff700 1 -- 172.21.15.78:0/17353623 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f61d4004d90 msgr2=0x7f61d4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:26.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.277+0000 7f61e3fff700 1 --2- 172.21.15.78:0/17353623 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f61d4004d90 0x7f61d4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:26.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.277+0000 7f61e3fff700 1 -- 172.21.15.78:0/17353623 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d40035b0 con 0x7f61d4003fb0 2023-10-14T11:59:26.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.277+0000 7f61e8f17700 1 --2- 172.21.15.78:0/17353623 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f61e40c1b70 0x7f61d4003a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:59:26.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.277+0000 7f61e9718700 1 --2- 172.21.15.78:0/17353623 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f61d4004d90 0x7f61d4005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:59:26.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.277+0000 7f61e3fff700 1 --2- 172.21.15.78:0/17353623 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61d4003fb0 0x7f61d4004850 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f61d0009b60 tx=0x7f61d4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82fc8a8697e53048 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:26.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.277+0000 7f61e37fe700 1 -- 172.21.15.78:0/17353623 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d000e6c0 con 0x7f61d4003fb0 2023-10-14T11:59:26.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.277+0000 7f61e37fe700 1 -- 172.21.15.78:0/17353623 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61d000e820 con 0x7f61d4003fb0 2023-10-14T11:59:26.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.277+0000 7f61e37fe700 1 -- 172.21.15.78:0/17353623 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d000eb90 con 0x7f61d4003fb0 2023-10-14T11:59:26.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.278+0000 7f61eb17b700 1 -- 172.21.15.78:0/17353623 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61d4003fb0 msgr2=0x7f61d4004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:26.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.278+0000 7f61eb17b700 1 --2- 172.21.15.78:0/17353623 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61d4003fb0 0x7f61d4004850 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f61d0009b60 tx=0x7f61d4005880 comp rx=0 tx=0).stop 2023-10-14T11:59:26.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.278+0000 7f61e9718700 1 -- 172.21.15.78:0/17353623 reap_dead start 2023-10-14T11:59:26.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.278+0000 7f61eb17b700 1 -- 172.21.15.78:0/17353623 shutdown_connections 2023-10-14T11:59:26.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.278+0000 7f61eb17b700 1 -- 172.21.15.78:0/17353623 >> 172.21.15.78:0/17353623 conn(0x7f61e40a8410 msgr2=0x7f61e40a8820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:26.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.278+0000 7f61eb17b700 1 -- 172.21.15.78:0/17353623 shutdown_connections 2023-10-14T11:59:26.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.278+0000 7f61eb17b700 1 -- 172.21.15.78:0/17353623 wait complete. 2023-10-14T11:59:26.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.279+0000 7f61eb17b700 1 Processor -- start 2023-10-14T11:59:26.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.279+0000 7f61eb17b700 1 -- start start 2023-10-14T11:59:26.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.280+0000 7f61eb17b700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f61d4003fb0 0x7f61e40c5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:26.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.280+0000 7f61eb17b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f61e41d71f0 0x7f61e41d7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:26.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.280+0000 7f61eb17b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61e41d64a0 0x7f61e41d6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:26.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.280+0000 7f61eb17b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e40c0040 con 0x7f61e41d64a0 2023-10-14T11:59:26.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.280+0000 7f61eb17b700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e41ed0a0 con 0x7f61e41d71f0 2023-10-14T11:59:26.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.280+0000 7f61eb17b700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61e41ed380 con 0x7f61d4003fb0 2023-10-14T11:59:26.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.280+0000 7f61e9718700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61e41d64a0 0x7f61e41d6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:26.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.280+0000 7f61e9718700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61e41d64a0 0x7f61e41d6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46622/0 (socket says 172.21.15.78:46622) 2023-10-14T11:59:26.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.280+0000 7f61e9718700 1 -- 172.21.15.78:0/830980603 learned_addr learned my addr 172.21.15.78:0/830980603 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:26.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.280+0000 7f61e8f17700 1 --2- 172.21.15.78:0/830980603 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f61d4003fb0 0x7f61e40c5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:26.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.280+0000 7f61e3fff700 1 --2- 172.21.15.78:0/830980603 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f61e41d71f0 0x7f61e41d7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:26.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.280+0000 7f61e9718700 1 -- 172.21.15.78:0/830980603 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f61d4003fb0 msgr2=0x7f61e40c5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:26.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.280+0000 7f61e9718700 1 --2- 172.21.15.78:0/830980603 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f61d4003fb0 0x7f61e40c5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:26.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.280+0000 7f61e9718700 1 -- 172.21.15.78:0/830980603 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f61e41d71f0 msgr2=0x7f61e41d7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:26.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.280+0000 7f61e9718700 1 --2- 172.21.15.78:0/830980603 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f61e41d71f0 0x7f61e41d7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:26.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.280+0000 7f61e9718700 1 -- 172.21.15.78:0/830980603 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e41ed660 con 0x7f61e41d64a0 2023-10-14T11:59:26.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.280+0000 7f61e3fff700 1 --2- 172.21.15.78:0/830980603 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f61e41d71f0 0x7f61e41d7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:59:26.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.280+0000 7f61e9718700 1 --2- 172.21.15.78:0/830980603 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61e41d64a0 0x7f61e41d6920 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f61e40a83b0 tx=0x7f61dc00de70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:26.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.280+0000 7f61e17fa700 1 -- 172.21.15.78:0/830980603 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61dc003e10 con 0x7f61e41d64a0 2023-10-14T11:59:26.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.280+0000 7f61e17fa700 1 -- 172.21.15.78:0/830980603 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61dc005070 con 0x7f61e41d64a0 2023-10-14T11:59:26.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.280+0000 7f61e17fa700 1 -- 172.21.15.78:0/830980603 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61dc005460 con 0x7f61e41d64a0 2023-10-14T11:59:26.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.280+0000 7f61eb17b700 1 -- 172.21.15.78:0/830980603 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61e41ed940 con 0x7f61e41d64a0 2023-10-14T11:59:26.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.280+0000 7f61eb17b700 1 -- 172.21.15.78:0/830980603 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e40c6150 con 0x7f61e41d64a0 2023-10-14T11:59:26.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.282+0000 7f61e17fa700 1 -- 172.21.15.78:0/830980603 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f61dc0055c0 con 0x7f61e41d64a0 2023-10-14T11:59:26.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.282+0000 7f61e17fa700 1 --2- 172.21.15.78:0/830980603 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f61c806af00 0x7f61c806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:26.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.282+0000 7f61e17fa700 1 -- 172.21.15.78:0/830980603 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f61dc089220 con 0x7f61e41d64a0 2023-10-14T11:59:26.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.282+0000 7f61e8f17700 1 --2- 172.21.15.78:0/830980603 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f61c806af00 0x7f61c806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:26.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.282+0000 7f61e8f17700 1 --2- 172.21.15.78:0/830980603 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f61c806af00 0x7f61c806d3c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f61d8001eb0 tx=0x7f61d80014d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:26.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.284+0000 7f61f7365b80 1 Processor -- start 2023-10-14T11:59:26.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.284+0000 7f61f7365b80 1 -- start start 2023-10-14T11:59:26.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.285+0000 7f61f7365b80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55636a4f96b0 0x55636a537620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:26.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.285+0000 7f61f7365b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55636a4f9a90 0x55636a537b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:26.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.285+0000 7f61f7365b80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55636a4f3c90 0x55636a5178a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:26.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.285+0000 7f61f7365b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55636a46dc20 con 0x55636a4f9a90 2023-10-14T11:59:26.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.285+0000 7f61f7365b80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x55636a452c10 con 0x55636a4f3c90 2023-10-14T11:59:26.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.285+0000 7f61f7365b80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x55636a450530 con 0x55636a4f96b0 2023-10-14T11:59:26.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.285+0000 7f61e3fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55636a4f9a90 0x55636a537b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:26.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.285+0000 7f61e3fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55636a4f9a90 0x55636a537b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46628/0 (socket says 172.21.15.78:46628) 2023-10-14T11:59:26.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.285+0000 7f61e3fff700 1 -- 172.21.15.78:0/3711232720 learned_addr learned my addr 172.21.15.78:0/3711232720 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:26.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.285+0000 7f61e3fff700 1 -- 172.21.15.78:0/3711232720 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55636a4f96b0 msgr2=0x55636a537620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:26.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.285+0000 7f61e3fff700 1 --2- 172.21.15.78:0/3711232720 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55636a4f96b0 0x55636a537620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:26.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.285+0000 7f61e3fff700 1 -- 172.21.15.78:0/3711232720 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55636a4f3c90 msgr2=0x55636a5178a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:26.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.285+0000 7f61e3fff700 1 --2- 172.21.15.78:0/3711232720 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55636a4f3c90 0x55636a5178a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:26.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.285+0000 7f61e3fff700 1 -- 172.21.15.78:0/3711232720 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55636a4093a0 con 0x55636a4f9a90 2023-10-14T11:59:26.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.286+0000 7f61e3fff700 1 --2- 172.21.15.78:0/3711232720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55636a4f9a90 0x55636a537b60 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f61d0016000 tx=0x7f61d0013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79ce5e074fc6c9db server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:26.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.286+0000 7f61ce7fc700 1 -- 172.21.15.78:0/3711232720 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d0021070 con 0x55636a4f9a90 2023-10-14T11:59:26.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.286+0000 7f61ce7fc700 1 -- 172.21.15.78:0/3711232720 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61d000e990 con 0x55636a4f9a90 2023-10-14T11:59:26.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.286+0000 7f61ce7fc700 1 -- 172.21.15.78:0/3711232720 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d000e6c0 con 0x55636a4f9a90 2023-10-14T11:59:26.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.286+0000 7f61f7365b80 1 -- 172.21.15.78:0/3711232720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55636a4f9a90 msgr2=0x55636a537b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:26.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.286+0000 7f61f7365b80 1 --2- 172.21.15.78:0/3711232720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55636a4f9a90 0x55636a537b60 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f61d0016000 tx=0x7f61d0013000 comp rx=0 tx=0).stop 2023-10-14T11:59:26.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.286+0000 7f61f7365b80 1 -- 172.21.15.78:0/3711232720 shutdown_connections 2023-10-14T11:59:26.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.286+0000 7f61f7365b80 1 --2- 172.21.15.78:0/3711232720 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55636a4f3c90 0x55636a5178a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:26.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.286+0000 7f61f7365b80 1 --2- 172.21.15.78:0/3711232720 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55636a4f96b0 0x55636a537620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:26.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.286+0000 7f61f7365b80 1 --2- 172.21.15.78:0/3711232720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55636a4f9a90 0x55636a537b60 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:26.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.286+0000 7f61f7365b80 1 -- 172.21.15.78:0/3711232720 >> 172.21.15.78:0/3711232720 conn(0x55636a426d80 msgr2=0x55636a4f73a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:26.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.286+0000 7f61f7365b80 1 -- 172.21.15.78:0/3711232720 shutdown_connections 2023-10-14T11:59:26.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.286+0000 7f61f7365b80 1 -- 172.21.15.78:0/3711232720 wait complete. 2023-10-14T11:59:26.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.287+0000 7f61f7365b80 1 -- start start 2023-10-14T11:59:26.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.287+0000 7f61f7365b80 1 Processor -- start 2023-10-14T11:59:26.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.288+0000 7f61f7365b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55636a4f3c90 0x55636a515ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:26.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.288+0000 7f61f7365b80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55636a4f96b0 0x55636a5112b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:26.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.288+0000 7f61f7365b80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55636a4f9a90 0x55636a5482d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:26.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.288+0000 7f61f7365b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55636a44be20 con 0x55636a4f3c90 2023-10-14T11:59:26.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.288+0000 7f61f7365b80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x55636a3e86c0 con 0x55636a4f96b0 2023-10-14T11:59:26.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.288+0000 7f61f7365b80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x55636a3e8360 con 0x55636a4f9a90 2023-10-14T11:59:26.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.288+0000 7f61f7365b80 10 client.0 _ll_register_callbacks cb 0x55636a2457b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T11:59:26.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.288+0000 7f61e8f17700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55636a4f3c90 0x55636a515ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:26.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.288+0000 7f61e8f17700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55636a4f3c90 0x55636a515ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46632/0 (socket says 172.21.15.78:46632) 2023-10-14T11:59:26.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.288+0000 7f61e8f17700 1 -- 172.21.15.78:0/2392142702 learned_addr learned my addr 172.21.15.78:0/2392142702 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:26.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.288+0000 7f61e9718700 1 --2- 172.21.15.78:0/2392142702 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55636a4f9a90 0x55636a5482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:26.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.288+0000 7f61e8f17700 1 -- 172.21.15.78:0/2392142702 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55636a4f9a90 msgr2=0x55636a5482d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:26.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.288+0000 7f61e8f17700 1 --2- 172.21.15.78:0/2392142702 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55636a4f9a90 0x55636a5482d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:26.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.288+0000 7f61e8f17700 1 -- 172.21.15.78:0/2392142702 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55636a4f96b0 msgr2=0x55636a5112b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:26.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.288+0000 7f61e3fff700 1 --2- 172.21.15.78:0/2392142702 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55636a4f96b0 0x55636a5112b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:26.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.288+0000 7f61e8f17700 1 --2- 172.21.15.78:0/2392142702 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55636a4f96b0 0x55636a5112b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:26.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.288+0000 7f61e8f17700 1 -- 172.21.15.78:0/2392142702 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61d40035b0 con 0x55636a4f3c90 2023-10-14T11:59:26.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.288+0000 7f61e3fff700 1 --2- 172.21.15.78:0/2392142702 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55636a4f96b0 0x55636a5112b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:59:26.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.288+0000 7f61e8f17700 1 --2- 172.21.15.78:0/2392142702 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55636a4f3c90 0x55636a515ee0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f61d80055d0 tx=0x7f61d80056b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af290a4fa2266a57 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:26.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.288+0000 7f61c37fe700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-14T11:59:26.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.289+0000 7f61c37fe700 1 -- 172.21.15.78:0/2392142702 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d80049b0 con 0x55636a4f3c90 2023-10-14T11:59:26.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.289+0000 7f61c37fe700 1 -- 172.21.15.78:0/2392142702 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f61d80057d0 con 0x55636a4f3c90 2023-10-14T11:59:26.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.289+0000 7f61f7365b80 1 -- 172.21.15.78:0/2392142702 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55636a4ec5b0 con 0x55636a4f3c90 2023-10-14T11:59:26.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.289+0000 7f61c37fe700 1 -- 172.21.15.78:0/2392142702 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f61d800fcb0 con 0x55636a4f3c90 2023-10-14T11:59:26.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.289+0000 7f61f7365b80 10 client.5316 fetch_fsmap learned FSMap version 17 2023-10-14T11:59:26.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.289+0000 7f61f7365b80 1 -- 172.21.15.78:0/2392142702 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55636a276d90 con 0x55636a4f3c90 2023-10-14T11:59:26.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.289+0000 7f61c37fe700 1 -- 172.21.15.78:0/2392142702 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55636a4ec5b0 con 0x55636a4f3c90 2023-10-14T11:59:26.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.289+0000 7f61c37fe700 1 -- 172.21.15.78:0/2392142702 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 17) v1 ==== 3299+0+0 (secure 0 0 0) 0x7f61d80213c0 con 0x55636a4f3c90 2023-10-14T11:59:26.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.289+0000 7f61f7365b80 10 client.5316 fetch_fsmap finished waiting for FSMap version 17 2023-10-14T11:59:26.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.289+0000 7f61f7365b80 10 client.5316 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-14T11:59:26.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.289+0000 7f61f7365b80 20 client.5316 populate_metadata read hostname 'smithi078' 2023-10-14T11:59:26.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.289+0000 7f61f7365b80 1 --2- 172.21.15.78:0/2392142702 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x55636a5164a0 0x55636a5345a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:26.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.289+0000 7f61f7365b80 4 client.5316 mds_command: new command op to 5031 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-14T11:59:26.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.289+0000 7f61f7365b80 1 -- 172.21.15.78:0/2392142702 --> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55636a444e00 con 0x55636a5164a0 2023-10-14T11:59:26.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.289+0000 7f61e3fff700 1 --2- 172.21.15.78:0/2392142702 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x55636a5164a0 0x55636a5345a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:26.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.290+0000 7f61c37fe700 10 client.5316 ms_handle_connect on v2:172.21.15.195:6833/2130671508 2023-10-14T11:59:26.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.290+0000 7f61e3fff700 1 --2- 172.21.15.78:0/2392142702 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x55636a5164a0 0x55636a5345a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88d3d8a5f51be9e1 server_cookie=c8db6ef6d9399545 in_seq=0 out_seq=0 2023-10-14T11:59:26.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.290+0000 7f61c37fe700 1 -- 172.21.15.78:0/2392142702 <== mds.0 v2:172.21.15.195:6833/2130671508 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55636a444e00 con 0x55636a5164a0 2023-10-14T11:59:26.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.290+0000 7f61c37fe700 10 client.5316 handle_command_reply: tid=0 2023-10-14T11:59:26.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.290+0000 7f61f7365b80 1 client.5316 shutdown 2023-10-14T11:59:26.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.290+0000 7f61f7365b80 1 -- 172.21.15.78:0/2392142702 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55636a4f3c90 msgr2=0x55636a515ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:26.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.290+0000 7f61f7365b80 1 --2- 172.21.15.78:0/2392142702 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55636a4f3c90 0x55636a515ee0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f61d80055d0 tx=0x7f61d80056b0 comp rx=0 tx=0).stop 2023-10-14T11:59:26.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.290+0000 7f61f7365b80 1 -- 172.21.15.78:0/2392142702 shutdown_connections 2023-10-14T11:59:26.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.290+0000 7f61f7365b80 1 --2- 172.21.15.78:0/2392142702 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55636a4f9a90 0x55636a5482d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:26.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.290+0000 7f61f7365b80 1 --2- 172.21.15.78:0/2392142702 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55636a4f3c90 0x55636a515ee0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:26.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.290+0000 7f61f7365b80 1 --2- 172.21.15.78:0/2392142702 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x55636a5164a0 0x55636a5345a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:26.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.290+0000 7f61f7365b80 1 --2- 172.21.15.78:0/2392142702 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55636a4f96b0 0x55636a5112b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:26.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.291+0000 7f61c37fe700 0 client.5316 ms_handle_reset on v2:172.21.15.195:6833/2130671508 2023-10-14T11:59:26.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.291+0000 7f61f7365b80 1 -- 172.21.15.78:0/2392142702 >> 172.21.15.78:0/2392142702 conn(0x55636a426d80 msgr2=0x55636a4ee7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:26.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.291+0000 7f61f7365b80 1 -- 172.21.15.78:0/2392142702 shutdown_connections 2023-10-14T11:59:26.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.291+0000 7f61f7365b80 1 -- 172.21.15.78:0/2392142702 wait complete. 2023-10-14T11:59:26.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.291+0000 7f61f7365b80 20 client.5316 trim_cache size 0 max 16384 2023-10-14T11:59:26.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.308+0000 7f61f7365b80 1 Processor -- start 2023-10-14T11:59:26.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.309+0000 7f61f7365b80 1 -- start start 2023-10-14T11:59:26.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.309+0000 7f61f7365b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55636a4f3c90 0x55636a5187c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:26.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.309+0000 7f61f7365b80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55636a4f96b0 0x55636a515830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:26.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.309+0000 7f61f7365b80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55636a4f9a90 0x55636a515d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:26.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.309+0000 7f61f7365b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55636a2cc8d0 con 0x55636a4f3c90 2023-10-14T11:59:26.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.309+0000 7f61f7365b80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x55636a4f41e0 con 0x55636a4f96b0 2023-10-14T11:59:26.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.309+0000 7f61f7365b80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x55636a536100 con 0x55636a4f9a90 2023-10-14T11:59:26.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.309+0000 7f61e8f17700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55636a4f3c90 0x55636a5187c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:26.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.309+0000 7f61e8f17700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55636a4f3c90 0x55636a5187c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46640/0 (socket says 172.21.15.78:46640) 2023-10-14T11:59:26.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.309+0000 7f61e8f17700 1 -- 172.21.15.78:0/3800077150 learned_addr learned my addr 172.21.15.78:0/3800077150 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:26.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.310+0000 7f61e8f17700 1 -- 172.21.15.78:0/3800077150 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55636a4f9a90 msgr2=0x55636a515d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:26.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.310+0000 7f61e8f17700 1 --2- 172.21.15.78:0/3800077150 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55636a4f9a90 0x55636a515d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:26.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.310+0000 7f61e8f17700 1 -- 172.21.15.78:0/3800077150 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55636a4f96b0 msgr2=0x55636a515830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:26.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.310+0000 7f61e8f17700 1 --2- 172.21.15.78:0/3800077150 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55636a4f96b0 0x55636a515830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:26.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.310+0000 7f61e8f17700 1 -- 172.21.15.78:0/3800077150 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e41d8dd0 con 0x55636a4f3c90 2023-10-14T11:59:26.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.310+0000 7f61e8f17700 1 --2- 172.21.15.78:0/3800077150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55636a4f3c90 0x55636a5187c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f61d800f4f0 tx=0x7f61d800a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=512a87cc39c9fed2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:26.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.310+0000 7f61ceffd700 1 -- 172.21.15.78:0/3800077150 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d80057d0 con 0x55636a4f3c90 2023-10-14T11:59:26.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.310+0000 7f61ceffd700 1 -- 172.21.15.78:0/3800077150 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61d800d070 con 0x55636a4f3c90 2023-10-14T11:59:26.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.310+0000 7f61ceffd700 1 -- 172.21.15.78:0/3800077150 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d8005930 con 0x55636a4f3c90 2023-10-14T11:59:26.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.310+0000 7f61f7365b80 1 -- 172.21.15.78:0/3800077150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55636a4f3c90 msgr2=0x55636a5187c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:26.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.310+0000 7f61f7365b80 1 --2- 172.21.15.78:0/3800077150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55636a4f3c90 0x55636a5187c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f61d800f4f0 tx=0x7f61d800a000 comp rx=0 tx=0).stop 2023-10-14T11:59:26.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.310+0000 7f61f7365b80 1 -- 172.21.15.78:0/3800077150 shutdown_connections 2023-10-14T11:59:26.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.310+0000 7f61f7365b80 1 --2- 172.21.15.78:0/3800077150 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55636a4f9a90 0x55636a515d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:26.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.311+0000 7f61f7365b80 1 --2- 172.21.15.78:0/3800077150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55636a4f3c90 0x55636a5187c0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:26.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.311+0000 7f61f7365b80 1 --2- 172.21.15.78:0/3800077150 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55636a4f96b0 0x55636a515830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:26.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.311+0000 7f61f7365b80 1 -- 172.21.15.78:0/3800077150 >> 172.21.15.78:0/3800077150 conn(0x55636a426d80 msgr2=0x55636a4f73a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:26.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.311+0000 7f61f7365b80 1 -- 172.21.15.78:0/3800077150 shutdown_connections 2023-10-14T11:59:26.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.311+0000 7f61f7365b80 1 -- 172.21.15.78:0/3800077150 wait complete. 2023-10-14T11:59:26.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.311+0000 7f61f7365b80 1 -- start start 2023-10-14T11:59:26.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.311+0000 7f61f7365b80 1 Processor -- start 2023-10-14T11:59:26.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.312+0000 7f61f7365b80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55636a4f3c90 0x55636a54f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:26.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.312+0000 7f61f7365b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55636a4f96b0 0x55636a4f8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:26.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.312+0000 7f61f7365b80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55636a4f9a90 0x55636a5a0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:26.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.312+0000 7f61f7365b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55636a54f8e0 con 0x55636a4f96b0 2023-10-14T11:59:26.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.312+0000 7f61f7365b80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x55636a512020 con 0x55636a4f3c90 2023-10-14T11:59:26.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.312+0000 7f61f7365b80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x55636a512830 con 0x55636a4f9a90 2023-10-14T11:59:26.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.312+0000 7f61f7365b80 10 client.0 _ll_register_callbacks cb 0x7f61d8004e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T11:59:26.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.312+0000 7f61e3fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55636a4f96b0 0x55636a4f8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:26.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.312+0000 7f61e3fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55636a4f96b0 0x55636a4f8b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46648/0 (socket says 172.21.15.78:46648) 2023-10-14T11:59:26.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.312+0000 7f61e3fff700 1 -- 172.21.15.78:0/1362221019 learned_addr learned my addr 172.21.15.78:0/1362221019 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:26.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.312+0000 7f61e8f17700 1 --2- 172.21.15.78:0/1362221019 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55636a4f3c90 0x55636a54f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:26.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.312+0000 7f61e9718700 1 --2- 172.21.15.78:0/1362221019 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55636a4f9a90 0x55636a5a0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:26.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.312+0000 7f61e3fff700 1 -- 172.21.15.78:0/1362221019 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55636a4f9a90 msgr2=0x55636a5a0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:26.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.312+0000 7f61e3fff700 1 --2- 172.21.15.78:0/1362221019 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55636a4f9a90 0x55636a5a0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:26.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.312+0000 7f61e3fff700 1 -- 172.21.15.78:0/1362221019 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55636a4f3c90 msgr2=0x55636a54f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:26.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.312+0000 7f61e3fff700 1 --2- 172.21.15.78:0/1362221019 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55636a4f3c90 0x55636a54f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:26.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.312+0000 7f61e3fff700 1 -- 172.21.15.78:0/1362221019 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55636a276d90 con 0x55636a4f96b0 2023-10-14T11:59:26.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.312+0000 7f61e8f17700 1 --2- 172.21.15.78:0/1362221019 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55636a4f3c90 0x55636a54f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:59:26.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.312+0000 7f61e9718700 1 --2- 172.21.15.78:0/1362221019 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55636a4f9a90 0x55636a5a0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:59:26.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.312+0000 7f61e3fff700 1 --2- 172.21.15.78:0/1362221019 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55636a4f96b0 0x55636a4f8b00 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f61d000e4b0 tx=0x7f61d00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7f04697bf974bde server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:26.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.312+0000 7f61c37fe700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-14T11:59:26.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.313+0000 7f61c37fe700 1 -- 172.21.15.78:0/1362221019 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d0021040 con 0x55636a4f96b0 2023-10-14T11:59:26.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.313+0000 7f61c37fe700 1 -- 172.21.15.78:0/1362221019 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f61d0004510 con 0x55636a4f96b0 2023-10-14T11:59:26.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.313+0000 7f61f7365b80 1 -- 172.21.15.78:0/1362221019 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f61d000e990 con 0x55636a4f96b0 2023-10-14T11:59:26.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.313+0000 7f61c37fe700 1 -- 172.21.15.78:0/1362221019 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f61d0005a20 con 0x55636a4f96b0 2023-10-14T11:59:26.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.313+0000 7f61f7365b80 10 client.5322 fetch_fsmap learned FSMap version 17 2023-10-14T11:59:26.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.313+0000 7f61f7365b80 1 -- 172.21.15.78:0/1362221019 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55636a25fc10 con 0x55636a4f96b0 2023-10-14T11:59:26.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.313+0000 7f61c37fe700 1 -- 172.21.15.78:0/1362221019 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61d000e990 con 0x55636a4f96b0 2023-10-14T11:59:26.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.313+0000 7f61c37fe700 1 -- 172.21.15.78:0/1362221019 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 17) v1 ==== 3299+0+0 (secure 0 0 0) 0x7f61d000e1d0 con 0x55636a4f96b0 2023-10-14T11:59:26.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.313+0000 7f61f7365b80 10 client.5322 fetch_fsmap finished waiting for FSMap version 17 2023-10-14T11:59:26.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.313+0000 7f61f7365b80 10 client.5322 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-14T11:59:26.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.313+0000 7f61f7365b80 20 client.5322 populate_metadata read hostname 'smithi078' 2023-10-14T11:59:26.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.313+0000 7f61f7365b80 1 --2- 172.21.15.78:0/1362221019 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x55636a5173f0 0x55636a5177d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:26.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.313+0000 7f61f7365b80 4 client.5322 mds_command: new command op to 5031 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-14T11:59:26.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.313+0000 7f61f7365b80 1 -- 172.21.15.78:0/1362221019 --> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55636a444b20 con 0x55636a5173f0 2023-10-14T11:59:26.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.313+0000 7f61e8f17700 1 --2- 172.21.15.78:0/1362221019 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x55636a5173f0 0x55636a5177d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:26.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.314+0000 7f61c37fe700 10 client.5322 ms_handle_connect on v2:172.21.15.195:6833/2130671508 2023-10-14T11:59:26.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.314+0000 7f61e8f17700 1 --2- 172.21.15.78:0/1362221019 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x55636a5173f0 0x55636a5177d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80595762d18cf92b server_cookie=ae1f595bdfcca864 in_seq=0 out_seq=0 2023-10-14T11:59:26.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.314+0000 7f61c37fe700 1 -- 172.21.15.78:0/1362221019 <== mds.0 v2:172.21.15.195:6833/2130671508 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+336 (crc 0 0 0) 0x55636a444b20 con 0x55636a5173f0 2023-10-14T11:59:26.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.314+0000 7f61c37fe700 10 client.5322 handle_command_reply: tid=0 2023-10-14T11:59:26.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.314+0000 7f61f7365b80 1 client.5322 shutdown 2023-10-14T11:59:26.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.314+0000 7f61f7365b80 1 -- 172.21.15.78:0/1362221019 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55636a4f96b0 msgr2=0x55636a4f8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:26.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.314+0000 7f61f7365b80 1 --2- 172.21.15.78:0/1362221019 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55636a4f96b0 0x55636a4f8b00 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f61d000e4b0 tx=0x7f61d00047c0 comp rx=0 tx=0).stop 2023-10-14T11:59:26.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.314+0000 7f61f7365b80 1 -- 172.21.15.78:0/1362221019 shutdown_connections 2023-10-14T11:59:26.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.314+0000 7f61f7365b80 1 --2- 172.21.15.78:0/1362221019 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55636a4f9a90 0x55636a5a0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:26.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.314+0000 7f61f7365b80 1 --2- 172.21.15.78:0/1362221019 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x55636a5173f0 0x55636a5177d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:26.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.314+0000 7f61f7365b80 1 --2- 172.21.15.78:0/1362221019 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55636a4f3c90 0x55636a54f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:26.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.314+0000 7f61c37fe700 0 client.5322 ms_handle_reset on v2:172.21.15.195:6833/2130671508 2023-10-14T11:59:26.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.314+0000 7f61f7365b80 1 --2- 172.21.15.78:0/1362221019 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55636a4f96b0 0x55636a4f8b00 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:26.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.314+0000 7f61f7365b80 1 -- 172.21.15.78:0/1362221019 >> 172.21.15.78:0/1362221019 conn(0x55636a426d80 msgr2=0x55636a539630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:26.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.314+0000 7f61f7365b80 1 -- 172.21.15.78:0/1362221019 shutdown_connections 2023-10-14T11:59:26.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.314+0000 7f61f7365b80 1 -- 172.21.15.78:0/1362221019 wait complete. 2023-10-14T11:59:26.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.314+0000 7f61f7365b80 20 client.5322 trim_cache size 0 max 16384 2023-10-14T11:59:26.345 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-14T11:59:26.345 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "4c5dc134-3092-406a-a1e1-13e6d9e926e8", 2023-10-14T11:59:26.345 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2023-10-14T11:59:26.345 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 5031, 2023-10-14T11:59:26.345 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2023-10-14T11:59:26.346 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2023-10-14T11:59:26.346 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "testcephfs1", 2023-10-14T11:59:26.346 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 24.130016571999999, 2023-10-14T11:59:26.346 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 17, 2023-10-14T11:59:26.347 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 32, 2023-10-14T11:59:26.347 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 32, 2023-10-14T11:59:26.347 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 32.020840573999998 2023-10-14T11:59:26.347 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-14T11:59:26.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.315+0000 7f61eb17b700 1 -- 172.21.15.78:0/830980603 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f61c806af00 msgr2=0x7f61c806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:26.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.315+0000 7f61eb17b700 1 --2- 172.21.15.78:0/830980603 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f61c806af00 0x7f61c806d3c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f61d8001eb0 tx=0x7f61d80014d0 comp rx=0 tx=0).stop 2023-10-14T11:59:26.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.315+0000 7f61eb17b700 1 -- 172.21.15.78:0/830980603 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61e41d64a0 msgr2=0x7f61e41d6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:26.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.315+0000 7f61eb17b700 1 --2- 172.21.15.78:0/830980603 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61e41d64a0 0x7f61e41d6920 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f61e40a83b0 tx=0x7f61dc00de70 comp rx=0 tx=0).stop 2023-10-14T11:59:26.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.315+0000 7f61eb17b700 1 -- 172.21.15.78:0/830980603 shutdown_connections 2023-10-14T11:59:26.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.315+0000 7f61eb17b700 1 --2- 172.21.15.78:0/830980603 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61e41d64a0 0x7f61e41d6920 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:26.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.315+0000 7f61eb17b700 1 --2- 172.21.15.78:0/830980603 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f61d4003fb0 0x7f61e40c5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:26.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.315+0000 7f61eb17b700 1 --2- 172.21.15.78:0/830980603 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f61c806af00 0x7f61c806d3c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:26.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.315+0000 7f61eb17b700 1 --2- 172.21.15.78:0/830980603 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f61e41d71f0 0x7f61e41d7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:26.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.315+0000 7f61eb17b700 1 -- 172.21.15.78:0/830980603 >> 172.21.15.78:0/830980603 conn(0x7f61e40a8410 msgr2=0x7f61e40a8820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:26.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.315+0000 7f61eb17b700 1 -- 172.21.15.78:0/830980603 shutdown_connections 2023-10-14T11:59:26.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:26.315+0000 7f61eb17b700 1 -- 172.21.15.78:0/830980603 wait complete. 2023-10-14T11:59:26.351 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 17, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-14T11:59:01.562764+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 17, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "testcephfs1", "in": [ 0 ], "info": { "gid_5031": { "addr": "172.21.15.195:6835/2130671508", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6833", "nonce": 2130671508, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2130671508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5031, "incarnation": 13, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2023-10-14T11:59:04.592935+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5031 } } } ], "standbys": [ { "addr": "172.21.15.195:6834/1492486786", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6832", "nonce": 1492486786, "type": "v2" }, { "addr": "172.21.15.195:6834", "nonce": 1492486786, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4658, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6836/2991384297", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6834", "nonce": 2991384297, "type": "v2" }, { "addr": "172.21.15.78:6836", "nonce": 2991384297, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5016, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6837/3246640037", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6835", "nonce": 3246640037, "type": "v2" }, { "addr": "172.21.15.78:6837", "nonce": 3246640037, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5019, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-14T11:59:26.353 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 2, 'name': 'b', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2023-10-14T11:59:26.353 INFO:tasks.cephfs.test_failover:current = [{'gid': 5019, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6837/3246640037', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 3246640037}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 3246640037}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 4658, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.195:6834/1492486786', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6832', 'nonce': 1492486786}, {'type': 'v1', 'addr': '172.21.15.195:6834', 'nonce': 1492486786}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 5016, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6836/2991384297', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 2991384297}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 2991384297}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 5031, 'name': 'd', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.195:6835/2130671508', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6833', 'nonce': 2130671508}, {'type': 'v1', 'addr': '172.21.15.195:6835', 'nonce': 2130671508}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2023-10-14T11:59:26.353 DEBUG:tasks.cephfs.test_failover:2 != -1 2023-10-14T11:59:26.354 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2023-10-14T11:59:31.355 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T11:59:31.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.493+0000 7f7e21a58700 1 Processor -- start 2023-10-14T11:59:31.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.493+0000 7f7e21a58700 1 -- start start 2023-10-14T11:59:31.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.493+0000 7f7e21a58700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e1c14b7e0 0x7f7e1c0c5150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:31.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.493+0000 7f7e21a58700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7e1c154b90 0x7f7e1c0c9260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:31.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.493+0000 7f7e21a58700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e1c0c4e40 con 0x7f7e1c0c97a0 2023-10-14T11:59:31.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.493+0000 7f7e21a58700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e1c0c81c0 con 0x7f7e1c154b90 2023-10-14T11:59:31.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.493+0000 7f7e21a58700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c0c84a0 con 0x7f7e1c14b7e0 2023-10-14T11:59:31.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.493+0000 7f7e1a7fc700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7e1c154b90 0x7f7e1c0c9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:31.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.494+0000 7f7e1b7fe700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f7e1c0c97a0 0x7f7e1c06b440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51532/0 (socket says 172.21.15.78:51532) 2023-10-14T11:59:31.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.494+0000 7f7e1b7fe700 1 -- 172.21.15.78:0/3867992709 learned_addr learned my addr 172.21.15.78:0/3867992709 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:59:31.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.494+0000 7f7e1affd700 1 --2- 172.21.15.78:0/3867992709 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e1c14b7e0 0x7f7e1c0c5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:31.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.494+0000 7f7e1a7fc700 1 -- 172.21.15.78:0/3867992709 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e1c14b7e0 msgr2=0x7f7e1c0c5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:31.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.494+0000 7f7e1a7fc700 1 --2- 172.21.15.78:0/3867992709 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e1c14b7e0 0x7f7e1c0c5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:31.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.494+0000 7f7e1a7fc700 1 -- 172.21.15.78:0/3867992709 >> v1:172.21.15.78:6789/0 conn(0x7f7e1c0c97a0 legacy=0x7f7e1c06b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:31.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.494+0000 7f7e1a7fc700 1 -- 172.21.15.78:0/3867992709 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e1c0c8780 con 0x7f7e1c154b90 2023-10-14T11:59:31.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.494+0000 7f7e19ffb700 1 -- 172.21.15.78:0/3867992709 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3998047950 0 0) 0x7f7e1c0c4e40 con 0x7f7e1c0c97a0 2023-10-14T11:59:31.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.494+0000 7f7e1affd700 1 --2- 172.21.15.78:0/3867992709 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e1c14b7e0 0x7f7e1c0c5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:59:31.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.494+0000 7f7e1a7fc700 1 --2- 172.21.15.78:0/3867992709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e1c154b90 0x7f7e1c0c9260 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c009840 tx=0x7f7e0c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afa8c29fa262a1fa server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:31.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.494+0000 7f7e19ffb700 1 -- 172.21.15.78:0/3867992709 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0c011830 con 0x7f7e1c154b90 2023-10-14T11:59:31.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.494+0000 7f7e19ffb700 1 -- 172.21.15.78:0/3867992709 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e0c011990 con 0x7f7e1c154b90 2023-10-14T11:59:31.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.494+0000 7f7e19ffb700 1 -- 172.21.15.78:0/3867992709 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0c011d20 con 0x7f7e1c154b90 2023-10-14T11:59:31.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.495+0000 7f7e21a58700 1 -- 172.21.15.78:0/3867992709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e1c154b90 msgr2=0x7f7e1c0c9260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:31.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.495+0000 7f7e21a58700 1 --2- 172.21.15.78:0/3867992709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e1c154b90 0x7f7e1c0c9260 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c009840 tx=0x7f7e0c009b50 comp rx=0 tx=0).stop 2023-10-14T11:59:31.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.495+0000 7f7e21a58700 1 -- 172.21.15.78:0/3867992709 shutdown_connections 2023-10-14T11:59:31.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.495+0000 7f7e21a58700 1 --2- 172.21.15.78:0/3867992709 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e1c14b7e0 0x7f7e1c0c5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:31.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.495+0000 7f7e21a58700 1 --2- 172.21.15.78:0/3867992709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e1c154b90 0x7f7e1c0c9260 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:31.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.495+0000 7f7e21a58700 1 -- 172.21.15.78:0/3867992709 >> 172.21.15.78:0/3867992709 conn(0x7f7e1c0ca2c0 msgr2=0x7f7e1c0ca6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:31.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.495+0000 7f7e21a58700 1 -- 172.21.15.78:0/3867992709 shutdown_connections 2023-10-14T11:59:31.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.495+0000 7f7e21a58700 1 -- 172.21.15.78:0/3867992709 wait complete. 2023-10-14T11:59:31.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.496+0000 7f7e21a58700 1 Processor -- start 2023-10-14T11:59:31.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.496+0000 7f7e21a58700 1 -- start start 2023-10-14T11:59:31.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.497+0000 7f7e21a58700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e1c14b7e0 0x7f7e1c150ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:31.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.497+0000 7f7e21a58700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e1c154b90 0x7f7e1c158330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:31.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.497+0000 7f7e21a58700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e1c150ad0 0x7f7e1c0bdd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:31.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.497+0000 7f7e21a58700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c0be960 con 0x7f7e1c14b7e0 2023-10-14T11:59:31.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.497+0000 7f7e21a58700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c0bec40 con 0x7f7e1c150ad0 2023-10-14T11:59:31.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.497+0000 7f7e21a58700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c0bef20 con 0x7f7e1c154b90 2023-10-14T11:59:31.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.497+0000 7f7e1affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e1c14b7e0 0x7f7e1c150ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:31.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.497+0000 7f7e1affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e1c14b7e0 0x7f7e1c150ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46658/0 (socket says 172.21.15.78:46658) 2023-10-14T11:59:31.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.497+0000 7f7e1affd700 1 -- 172.21.15.78:0/375749329 learned_addr learned my addr 172.21.15.78:0/375749329 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:31.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.497+0000 7f7e1a7fc700 1 --2- 172.21.15.78:0/375749329 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e1c154b90 0x7f7e1c158330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:31.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.497+0000 7f7e1b7fe700 1 --2- 172.21.15.78:0/375749329 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e1c150ad0 0x7f7e1c0bdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:31.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.497+0000 7f7e1affd700 1 -- 172.21.15.78:0/375749329 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e1c154b90 msgr2=0x7f7e1c158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:31.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.497+0000 7f7e1affd700 1 --2- 172.21.15.78:0/375749329 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e1c154b90 0x7f7e1c158330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:31.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.497+0000 7f7e1affd700 1 -- 172.21.15.78:0/375749329 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e1c150ad0 msgr2=0x7f7e1c0bdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:31.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.497+0000 7f7e1affd700 1 --2- 172.21.15.78:0/375749329 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e1c150ad0 0x7f7e1c0bdd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:31.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.497+0000 7f7e1affd700 1 -- 172.21.15.78:0/375749329 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e0c0094a0 con 0x7f7e1c14b7e0 2023-10-14T11:59:31.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.497+0000 7f7e1a7fc700 1 --2- 172.21.15.78:0/375749329 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e1c154b90 0x7f7e1c158330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:59:31.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.498+0000 7f7e1affd700 1 --2- 172.21.15.78:0/375749329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e1c14b7e0 0x7f7e1c150ee0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f7e1000e810 tx=0x7f7e1000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:31.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.498+0000 7f7e03fff700 1 -- 172.21.15.78:0/375749329 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e10004230 con 0x7f7e1c14b7e0 2023-10-14T11:59:31.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.498+0000 7f7e03fff700 1 -- 172.21.15.78:0/375749329 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e10004390 con 0x7f7e1c14b7e0 2023-10-14T11:59:31.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.498+0000 7f7e21a58700 1 -- 172.21.15.78:0/375749329 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e1c0bf200 con 0x7f7e1c14b7e0 2023-10-14T11:59:31.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.498+0000 7f7e03fff700 1 -- 172.21.15.78:0/375749329 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e100036a0 con 0x7f7e1c14b7e0 2023-10-14T11:59:31.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.498+0000 7f7e21a58700 1 -- 172.21.15.78:0/375749329 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e1c158870 con 0x7f7e1c14b7e0 2023-10-14T11:59:31.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.499+0000 7f7e21a58700 1 -- 172.21.15.78:0/375749329 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e1c0a9a80 con 0x7f7e1c14b7e0 2023-10-14T11:59:31.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.499+0000 7f7e03fff700 1 -- 172.21.15.78:0/375749329 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7e10003800 con 0x7f7e1c14b7e0 2023-10-14T11:59:31.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.499+0000 7f7e03fff700 1 --2- 172.21.15.78:0/375749329 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7e0806af00 0x7f7e0806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:31.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.500+0000 7f7e03fff700 1 -- 172.21.15.78:0/375749329 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f7e10089060 con 0x7f7e1c14b7e0 2023-10-14T11:59:31.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.500+0000 7f7e1a7fc700 1 --2- 172.21.15.78:0/375749329 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7e0806af00 0x7f7e0806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:31.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.500+0000 7f7e1a7fc700 1 --2- 172.21.15.78:0/375749329 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7e0806af00 0x7f7e0806d3c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c009840 tx=0x7f7e0c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:31.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.504+0000 7f7e03fff700 1 -- 172.21.15.78:0/375749329 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7e10054180 con 0x7f7e1c14b7e0 2023-10-14T11:59:31.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.736+0000 7f7e21a58700 1 -- 172.21.15.78:0/375749329 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e1c0beaa0 con 0x7f7e1c14b7e0 2023-10-14T11:59:31.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.736+0000 7f7e03fff700 1 -- 172.21.15.78:0/375749329 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4924 (secure 0 0 0) 0x7f7e1c0beaa0 con 0x7f7e1c14b7e0 2023-10-14T11:59:31.739 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:59:31.739 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4658,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6834/1492486786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1492486786},{"type":"v1","addr":"172.21.15.195:6834","nonce":1492486786}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5016,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/2991384297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2991384297},{"type":"v1","addr":"172.21.15.78:6836","nonce":2991384297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5019,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3246640037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3246640037},{"type":"v1","addr":"172.21.15.78:6837","nonce":3246640037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T11:59:01.562764+0000","modified":"2023-10-14T11:59:04.592935+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5031":{"gid":5031,"name":"d","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.195:6835/2130671508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6833","nonce":2130671508},{"type":"v1","addr":"172.21.15.195:6835","nonce":2130671508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"testcephfs1","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-14T11:59:31.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.739+0000 7f7e21a58700 1 -- 172.21.15.78:0/375749329 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7e0806af00 msgr2=0x7f7e0806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:31.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.739+0000 7f7e21a58700 1 --2- 172.21.15.78:0/375749329 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7e0806af00 0x7f7e0806d3c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c009840 tx=0x7f7e0c008030 comp rx=0 tx=0).stop 2023-10-14T11:59:31.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.739+0000 7f7e21a58700 1 -- 172.21.15.78:0/375749329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e1c14b7e0 msgr2=0x7f7e1c150ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:31.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.739+0000 7f7e21a58700 1 --2- 172.21.15.78:0/375749329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e1c14b7e0 0x7f7e1c150ee0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f7e1000e810 tx=0x7f7e1000ebd0 comp rx=0 tx=0).stop 2023-10-14T11:59:31.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.739+0000 7f7e21a58700 1 -- 172.21.15.78:0/375749329 shutdown_connections 2023-10-14T11:59:31.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.739+0000 7f7e21a58700 1 --2- 172.21.15.78:0/375749329 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f7e0806af00 0x7f7e0806d3c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:31.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.739+0000 7f7e21a58700 1 --2- 172.21.15.78:0/375749329 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e1c150ad0 0x7f7e1c0bdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:31.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.739+0000 7f7e21a58700 1 --2- 172.21.15.78:0/375749329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e1c14b7e0 0x7f7e1c150ee0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:31.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.739+0000 7f7e21a58700 1 --2- 172.21.15.78:0/375749329 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e1c154b90 0x7f7e1c158330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:31.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.739+0000 7f7e21a58700 1 -- 172.21.15.78:0/375749329 >> 172.21.15.78:0/375749329 conn(0x7f7e1c0ca2c0 msgr2=0x7f7e1c148c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:31.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.739+0000 7f7e21a58700 1 -- 172.21.15.78:0/375749329 shutdown_connections 2023-10-14T11:59:31.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:31.739+0000 7f7e21a58700 1 -- 172.21.15.78:0/375749329 wait complete. 2023-10-14T11:59:31.744 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 17 2023-10-14T11:59:32.156 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-14T11:59:01.562764+0000', 'modified': '2023-10-14T11:59:04.592935+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5031}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5031': {'gid': 5031, 'name': 'd', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.195:6835/2130671508', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6833', 'nonce': 2130671508}, {'type': 'v1', 'addr': '172.21.15.195:6835', 'nonce': 2130671508}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'testcephfs1', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-14T11:59:32.156 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-14T11:59:32.157 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-14T11:59:32.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.277+0000 7fbb02153700 1 Processor -- start 2023-10-14T11:59:32.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.277+0000 7fbb02153700 1 -- start start 2023-10-14T11:59:32.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.278+0000 7fbb02153700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fbafc14ecc0 0x7fbafc151730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:32.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.278+0000 7fbb02153700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbafc06e080 con 0x7fbafc0c1c00 2023-10-14T11:59:32.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.278+0000 7fbb02153700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbafc06e360 con 0x7fbafc06da10 2023-10-14T11:59:32.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.278+0000 7fbb02153700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbafc06de70 con 0x7fbafc14ecc0 2023-10-14T11:59:32.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.278+0000 7fbafbfff700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fbafc0c1c00 0x7fbafc0c2050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51542/0 (socket says 172.21.15.78:51542) 2023-10-14T11:59:32.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.278+0000 7fbafbfff700 1 -- 172.21.15.78:0/285361007 learned_addr learned my addr 172.21.15.78:0/285361007 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:59:32.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.278+0000 7fbafaffd700 1 -- 172.21.15.78:0/285361007 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2656303112 0 0) 0x7fbafc06e080 con 0x7fbafc0c1c00 2023-10-14T11:59:32.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.278+0000 7fbafaffd700 1 -- 172.21.15.78:0/285361007 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbae80035b0 con 0x7fbafc0c1c00 2023-10-14T11:59:32.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.278+0000 7fbafb7fe700 1 --2- 172.21.15.78:0/285361007 >> v2:172.21.15.195:3300/0 conn(0x7fbafc14ecc0 0x7fbafc151730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:32.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.278+0000 7fbafaffd700 1 -- 172.21.15.78:0/285361007 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 928984320 0 0) 0x7fbae80035b0 con 0x7fbafc0c1c00 2023-10-14T11:59:32.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.278+0000 7fbafaffd700 1 -- 172.21.15.78:0/285361007 >> v2:172.21.15.195:3300/0 conn(0x7fbafc14ecc0 msgr2=0x7fbafc151730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:32.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.278+0000 7fbafaffd700 1 --2- 172.21.15.78:0/285361007 >> v2:172.21.15.195:3300/0 conn(0x7fbafc14ecc0 0x7fbafc151730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:32.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.278+0000 7fbafaffd700 1 -- 172.21.15.78:0/285361007 >> v1:172.21.15.195:6789/0 conn(0x7fbafc06da10 legacy=0x7fbafc06eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:32.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.278+0000 7fbafaffd700 1 -- 172.21.15.78:0/285361007 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbafc06e640 con 0x7fbafc0c1c00 2023-10-14T11:59:32.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.278+0000 7fbafaffd700 1 -- 172.21.15.78:0/285361007 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441799917 0 0) 0x7fbaf4001a40 con 0x7fbafc0c1c00 2023-10-14T11:59:32.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.279+0000 7fbafaffd700 1 -- 172.21.15.78:0/285361007 >> v1:172.21.15.78:6789/0 conn(0x7fbafc0c1c00 legacy=0x7fbafc0c2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:32.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.279+0000 7fbafaffd700 1 --2- 172.21.15.78:0/285361007 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbae8003a00 0x7fbae8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:32.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.279+0000 7fbafaffd700 1 --2- 172.21.15.78:0/285361007 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbae8004850 0x7fbae8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:32.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.279+0000 7fbafaffd700 1 --2- 172.21.15.78:0/285361007 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbae8005330 0x7fbae800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:32.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.279+0000 7fbafaffd700 1 -- 172.21.15.78:0/285361007 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc06de70 con 0x7fbae8004850 2023-10-14T11:59:32.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.279+0000 7fbafaffd700 1 -- 172.21.15.78:0/285361007 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae800c1e0 con 0x7fbae8005330 2023-10-14T11:59:32.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.279+0000 7fbafaffd700 1 -- 172.21.15.78:0/285361007 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbae800c3a0 con 0x7fbae8003a00 2023-10-14T11:59:32.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.279+0000 7fbaf2ffd700 1 --2- 172.21.15.78:0/285361007 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbae8004850 0x7fbae8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:32.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.279+0000 7fbafb7fe700 1 --2- 172.21.15.78:0/285361007 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbae8003a00 0x7fbae8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:32.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.279+0000 7fbafbfff700 1 --2- 172.21.15.78:0/285361007 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbae8005330 0x7fbae800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:32.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.279+0000 7fbaf2ffd700 1 -- 172.21.15.78:0/285361007 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbae8003a00 msgr2=0x7fbae8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:32.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.279+0000 7fbaf2ffd700 1 --2- 172.21.15.78:0/285361007 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbae8003a00 0x7fbae8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:32.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.279+0000 7fbaf2ffd700 1 -- 172.21.15.78:0/285361007 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbae8005330 msgr2=0x7fbae800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:32.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.279+0000 7fbaf2ffd700 1 --2- 172.21.15.78:0/285361007 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbae8005330 0x7fbae800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:32.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.279+0000 7fbafbfff700 1 -- 172.21.15.78:0/285361007 reap_dead start 2023-10-14T11:59:32.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.279+0000 7fbaf2ffd700 1 -- 172.21.15.78:0/285361007 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae80035b0 con 0x7fbae8004850 2023-10-14T11:59:32.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.279+0000 7fbaf2ffd700 1 --2- 172.21.15.78:0/285361007 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbae8004850 0x7fbae8004cd0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fbae4007fd0 tx=0x7fbae8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24a023ee88ae174f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:32.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.279+0000 7fbafaffd700 1 -- 172.21.15.78:0/285361007 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae4016070 con 0x7fbae8004850 2023-10-14T11:59:32.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.279+0000 7fbafaffd700 1 -- 172.21.15.78:0/285361007 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbae4006860 con 0x7fbae8004850 2023-10-14T11:59:32.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.279+0000 7fbafaffd700 1 -- 172.21.15.78:0/285361007 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae4003950 con 0x7fbae8004850 2023-10-14T11:59:32.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.280+0000 7fbb02153700 1 -- 172.21.15.78:0/285361007 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbae8004850 msgr2=0x7fbae8004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:32.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.280+0000 7fbb02153700 1 --2- 172.21.15.78:0/285361007 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbae8004850 0x7fbae8004cd0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fbae4007fd0 tx=0x7fbae8005880 comp rx=0 tx=0).stop 2023-10-14T11:59:32.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.280+0000 7fbb02153700 1 -- 172.21.15.78:0/285361007 shutdown_connections 2023-10-14T11:59:32.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.280+0000 7fbb02153700 1 --2- 172.21.15.78:0/285361007 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbae8004850 0x7fbae8004cd0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:32.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.280+0000 7fbb02153700 1 -- 172.21.15.78:0/285361007 >> 172.21.15.78:0/285361007 conn(0x7fbafc0c3a20 msgr2=0x7fbafc0c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:32.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.280+0000 7fbb02153700 1 -- 172.21.15.78:0/285361007 shutdown_connections 2023-10-14T11:59:32.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.280+0000 7fbb02153700 1 -- 172.21.15.78:0/285361007 wait complete. 2023-10-14T11:59:32.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.281+0000 7fbb02153700 1 Processor -- start 2023-10-14T11:59:32.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.281+0000 7fbb02153700 1 -- start start 2023-10-14T11:59:32.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.282+0000 7fbb02153700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbae8004850 0x7fbafc1552e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:32.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.282+0000 7fbb02153700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbafc06aab0 0x7fbafc1eda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:32.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.282+0000 7fbb02153700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbafc1edf50 0x7fbafc1ea420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:32.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.282+0000 7fbb02153700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc1e9fd0 con 0x7fbae8004850 2023-10-14T11:59:32.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.282+0000 7fbb02153700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc155b80 con 0x7fbafc06aab0 2023-10-14T11:59:32.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.282+0000 7fbb02153700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc1ed5c0 con 0x7fbafc1edf50 2023-10-14T11:59:32.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.282+0000 7fbafb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbae8004850 0x7fbafc1552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:32.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.282+0000 7fbafb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbae8004850 0x7fbafc1552e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46676/0 (socket says 172.21.15.78:46676) 2023-10-14T11:59:32.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.282+0000 7fbafb7fe700 1 -- 172.21.15.78:0/1890921233 learned_addr learned my addr 172.21.15.78:0/1890921233 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:32.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.282+0000 7fbafb7fe700 1 -- 172.21.15.78:0/1890921233 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbafc1edf50 msgr2=0x7fbafc1ea420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:32.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.282+0000 7fbafb7fe700 1 --2- 172.21.15.78:0/1890921233 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbafc1edf50 0x7fbafc1ea420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:32.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.282+0000 7fbafb7fe700 1 -- 172.21.15.78:0/1890921233 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbafc06aab0 msgr2=0x7fbafc1eda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:32.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.282+0000 7fbafb7fe700 1 --2- 172.21.15.78:0/1890921233 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbafc06aab0 0x7fbafc1eda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:32.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.282+0000 7fbafb7fe700 1 -- 172.21.15.78:0/1890921233 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbafc0abba0 con 0x7fbae8004850 2023-10-14T11:59:32.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.282+0000 7fbafb7fe700 1 --2- 172.21.15.78:0/1890921233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbae8004850 0x7fbafc1552e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fbaec010810 tx=0x7fbaec010bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:32.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.283+0000 7fbaf8ff9700 1 -- 172.21.15.78:0/1890921233 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaec004230 con 0x7fbae8004850 2023-10-14T11:59:32.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.283+0000 7fbaf8ff9700 1 -- 172.21.15.78:0/1890921233 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaec00c050 con 0x7fbae8004850 2023-10-14T11:59:32.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.283+0000 7fbaf8ff9700 1 -- 172.21.15.78:0/1890921233 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaec0045b0 con 0x7fbae8004850 2023-10-14T11:59:32.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.283+0000 7fbb02153700 1 -- 172.21.15.78:0/1890921233 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbafc0abe80 con 0x7fbae8004850 2023-10-14T11:59:32.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.283+0000 7fbb02153700 1 -- 172.21.15.78:0/1890921233 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbafc1ee3d0 con 0x7fbae8004850 2023-10-14T11:59:32.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.284+0000 7fbaf8ff9700 1 -- 172.21.15.78:0/1890921233 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fbaec0036a0 con 0x7fbae8004850 2023-10-14T11:59:32.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.285+0000 7fbaf8ff9700 1 --2- 172.21.15.78:0/1890921233 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fbadc06af00 0x7fbadc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:32.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.285+0000 7fbaf8ff9700 1 -- 172.21.15.78:0/1890921233 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7fbaec089d10 con 0x7fbae8004850 2023-10-14T11:59:32.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.285+0000 7fbaf2ffd700 1 --2- 172.21.15.78:0/1890921233 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fbadc06af00 0x7fbadc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:32.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.285+0000 7fbaf2ffd700 1 --2- 172.21.15.78:0/1890921233 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fbadc06af00 0x7fbadc06d3c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbafc154fd0 tx=0x7fbae4011000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:32.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.287+0000 7fbb0e33db80 1 Processor -- start 2023-10-14T11:59:32.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.287+0000 7fbb0e33db80 1 -- start start 2023-10-14T11:59:32.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.287+0000 7fbb0e33db80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55c5e77396b0 0x55c5e7777620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:32.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.287+0000 7fbb0e33db80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c5e7739a90 0x55c5e7777b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:32.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.287+0000 7fbb0e33db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c5e7733c90 0x55c5e77578a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:32.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.287+0000 7fbb0e33db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e76adc20 con 0x55c5e7733c90 2023-10-14T11:59:32.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.287+0000 7fbb0e33db80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e7692c10 con 0x55c5e7739a90 2023-10-14T11:59:32.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.287+0000 7fbb0e33db80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c5e7690530 con 0x55c5e77396b0 2023-10-14T11:59:32.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.287+0000 7fbafbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c5e7733c90 0x55c5e77578a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:32.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.288+0000 7fbafbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c5e7733c90 0x55c5e77578a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46686/0 (socket says 172.21.15.78:46686) 2023-10-14T11:59:32.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.288+0000 7fbafbfff700 1 -- 172.21.15.78:0/4068482373 learned_addr learned my addr 172.21.15.78:0/4068482373 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:32.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.288+0000 7fbafb7fe700 1 --2- 172.21.15.78:0/4068482373 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55c5e77396b0 0x55c5e7777620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:32.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.288+0000 7fbaf2ffd700 1 --2- 172.21.15.78:0/4068482373 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c5e7739a90 0x55c5e7777b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:32.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.288+0000 7fbafbfff700 1 -- 172.21.15.78:0/4068482373 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55c5e77396b0 msgr2=0x55c5e7777620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:32.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.288+0000 7fbafbfff700 1 --2- 172.21.15.78:0/4068482373 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55c5e77396b0 0x55c5e7777620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:32.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.288+0000 7fbafbfff700 1 -- 172.21.15.78:0/4068482373 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c5e7739a90 msgr2=0x55c5e7777b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:32.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.288+0000 7fbafbfff700 1 --2- 172.21.15.78:0/4068482373 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c5e7739a90 0x55c5e7777b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:32.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.288+0000 7fbafbfff700 1 -- 172.21.15.78:0/4068482373 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c5e76493a0 con 0x55c5e7733c90 2023-10-14T11:59:32.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.288+0000 7fbaf2ffd700 1 --2- 172.21.15.78:0/4068482373 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c5e7739a90 0x55c5e7777b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:59:32.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.288+0000 7fbafbfff700 1 --2- 172.21.15.78:0/4068482373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c5e7733c90 0x55c5e77578a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fbafc0c4890 tx=0x7fbaf4007330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=114a843aa1347bcc server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:32.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.288+0000 7fbaf17fa700 1 -- 172.21.15.78:0/4068482373 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf40079f0 con 0x55c5e7733c90 2023-10-14T11:59:32.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.288+0000 7fbaf17fa700 1 -- 172.21.15.78:0/4068482373 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf4007b50 con 0x55c5e7733c90 2023-10-14T11:59:32.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.288+0000 7fbaf17fa700 1 -- 172.21.15.78:0/4068482373 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf4014680 con 0x55c5e7733c90 2023-10-14T11:59:32.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.289+0000 7fbb0e33db80 1 -- 172.21.15.78:0/4068482373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c5e7733c90 msgr2=0x55c5e77578a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:32.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.289+0000 7fbb0e33db80 1 --2- 172.21.15.78:0/4068482373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c5e7733c90 0x55c5e77578a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fbafc0c4890 tx=0x7fbaf4007330 comp rx=0 tx=0).stop 2023-10-14T11:59:32.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.289+0000 7fbb0e33db80 1 -- 172.21.15.78:0/4068482373 shutdown_connections 2023-10-14T11:59:32.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.289+0000 7fbb0e33db80 1 --2- 172.21.15.78:0/4068482373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c5e7733c90 0x55c5e77578a0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:32.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.289+0000 7fbb0e33db80 1 --2- 172.21.15.78:0/4068482373 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55c5e77396b0 0x55c5e7777620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:32.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.289+0000 7fbb0e33db80 1 --2- 172.21.15.78:0/4068482373 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c5e7739a90 0x55c5e7777b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:32.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.289+0000 7fbb0e33db80 1 -- 172.21.15.78:0/4068482373 >> 172.21.15.78:0/4068482373 conn(0x55c5e7666d80 msgr2=0x55c5e77373a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:32.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.289+0000 7fbb0e33db80 1 -- 172.21.15.78:0/4068482373 shutdown_connections 2023-10-14T11:59:32.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.289+0000 7fbb0e33db80 1 -- 172.21.15.78:0/4068482373 wait complete. 2023-10-14T11:59:32.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.289+0000 7fbb0e33db80 1 -- start start 2023-10-14T11:59:32.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.290+0000 7fbb0e33db80 1 Processor -- start 2023-10-14T11:59:32.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.290+0000 7fbb0e33db80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55c5e7733c90 0x55c5e7782720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:32.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.290+0000 7fbb0e33db80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c5e77396b0 0x55c5e775d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:32.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.290+0000 7fbb0e33db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c5e7739a90 0x55c5e775c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:32.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.290+0000 7fbb0e33db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e768be20 con 0x55c5e7739a90 2023-10-14T11:59:32.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.290+0000 7fbb0e33db80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e76286c0 con 0x55c5e77396b0 2023-10-14T11:59:32.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.290+0000 7fbb0e33db80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c5e7628360 con 0x55c5e7733c90 2023-10-14T11:59:32.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.290+0000 7fbb0e33db80 10 client.0 _ll_register_callbacks cb 0x55c5e74857b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T11:59:32.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.290+0000 7fbafbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c5e7739a90 0x55c5e775c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:32.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.290+0000 7fbafbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c5e7739a90 0x55c5e775c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46692/0 (socket says 172.21.15.78:46692) 2023-10-14T11:59:32.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.290+0000 7fbafbfff700 1 -- 172.21.15.78:0/538236238 learned_addr learned my addr 172.21.15.78:0/538236238 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:32.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.291+0000 7fbafb7fe700 1 --2- 172.21.15.78:0/538236238 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55c5e7733c90 0x55c5e7782720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:32.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.291+0000 7fbaf2ffd700 1 --2- 172.21.15.78:0/538236238 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c5e77396b0 0x55c5e775d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:32.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.291+0000 7fbafbfff700 1 -- 172.21.15.78:0/538236238 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55c5e7733c90 msgr2=0x55c5e7782720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:32.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.291+0000 7fbafbfff700 1 --2- 172.21.15.78:0/538236238 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55c5e7733c90 0x55c5e7782720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:32.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.291+0000 7fbafbfff700 1 -- 172.21.15.78:0/538236238 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c5e77396b0 msgr2=0x55c5e775d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:32.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.291+0000 7fbafbfff700 1 --2- 172.21.15.78:0/538236238 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c5e77396b0 0x55c5e775d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:32.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.291+0000 7fbafbfff700 1 -- 172.21.15.78:0/538236238 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaf400f040 con 0x55c5e7739a90 2023-10-14T11:59:32.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.291+0000 7fbafbfff700 1 --2- 172.21.15.78:0/538236238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c5e7739a90 0x55c5e775c9a0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x55c5e7737f70 tx=0x7fbaf4002010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47fcc60ea86bfc21 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:32.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.291+0000 7fbada7fc700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-14T11:59:32.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.291+0000 7fbada7fc700 1 -- 172.21.15.78:0/538236238 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf40020b0 con 0x55c5e7739a90 2023-10-14T11:59:32.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.291+0000 7fbada7fc700 1 -- 172.21.15.78:0/538236238 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7fbaf401f410 con 0x55c5e7739a90 2023-10-14T11:59:32.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.291+0000 7fbb0e33db80 1 -- 172.21.15.78:0/538236238 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c5e772c5b0 con 0x55c5e7739a90 2023-10-14T11:59:32.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.291+0000 7fbada7fc700 1 -- 172.21.15.78:0/538236238 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbaf401f610 con 0x55c5e7739a90 2023-10-14T11:59:32.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.291+0000 7fbb0e33db80 10 client.5343 fetch_fsmap learned FSMap version 17 2023-10-14T11:59:32.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.291+0000 7fbb0e33db80 1 -- 172.21.15.78:0/538236238 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55c5e74b6d90 con 0x55c5e7739a90 2023-10-14T11:59:32.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.291+0000 7fbada7fc700 1 -- 172.21.15.78:0/538236238 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c5e772c5b0 con 0x55c5e7739a90 2023-10-14T11:59:32.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.291+0000 7fbada7fc700 1 -- 172.21.15.78:0/538236238 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 17) v1 ==== 3299+0+0 (secure 0 0 0) 0x7fbaf4025020 con 0x55c5e7739a90 2023-10-14T11:59:32.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.291+0000 7fbb0e33db80 10 client.5343 fetch_fsmap finished waiting for FSMap version 17 2023-10-14T11:59:32.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.291+0000 7fbb0e33db80 10 client.5343 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-14T11:59:32.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.291+0000 7fbb0e33db80 20 client.5343 populate_metadata read hostname 'smithi078' 2023-10-14T11:59:32.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.291+0000 7fbb0e33db80 1 --2- 172.21.15.78:0/538236238 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x55c5e775dda0 0x55c5e7788410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:32.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.291+0000 7fbb0e33db80 4 client.5343 mds_command: new command op to 5031 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-14T11:59:32.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.291+0000 7fbb0e33db80 1 -- 172.21.15.78:0/538236238 --> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c5e7684e00 con 0x55c5e775dda0 2023-10-14T11:59:32.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.292+0000 7fbafb7fe700 1 --2- 172.21.15.78:0/538236238 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x55c5e775dda0 0x55c5e7788410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:32.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.292+0000 7fbada7fc700 10 client.5343 ms_handle_connect on v2:172.21.15.195:6833/2130671508 2023-10-14T11:59:32.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.292+0000 7fbafb7fe700 1 --2- 172.21.15.78:0/538236238 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x55c5e775dda0 0x55c5e7788410 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=edaa9890ff6612d6 server_cookie=4fd0a15954e0c8d7 in_seq=0 out_seq=0 2023-10-14T11:59:32.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.293+0000 7fbada7fc700 1 -- 172.21.15.78:0/538236238 <== mds.0 v2:172.21.15.195:6833/2130671508 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c5e7684e00 con 0x55c5e775dda0 2023-10-14T11:59:32.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.293+0000 7fbada7fc700 10 client.5343 handle_command_reply: tid=0 2023-10-14T11:59:32.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.293+0000 7fbb0e33db80 1 client.5343 shutdown 2023-10-14T11:59:32.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.293+0000 7fbb0e33db80 1 -- 172.21.15.78:0/538236238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c5e7739a90 msgr2=0x55c5e775c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:32.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.293+0000 7fbb0e33db80 1 --2- 172.21.15.78:0/538236238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c5e7739a90 0x55c5e775c9a0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x55c5e7737f70 tx=0x7fbaf4002010 comp rx=0 tx=0).stop 2023-10-14T11:59:32.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.293+0000 7fbb0e33db80 1 -- 172.21.15.78:0/538236238 shutdown_connections 2023-10-14T11:59:32.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.293+0000 7fbb0e33db80 1 --2- 172.21.15.78:0/538236238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c5e7739a90 0x55c5e775c9a0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:32.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.293+0000 7fbb0e33db80 1 --2- 172.21.15.78:0/538236238 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55c5e7733c90 0x55c5e7782720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:32.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.293+0000 7fbb0e33db80 1 --2- 172.21.15.78:0/538236238 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x55c5e775dda0 0x55c5e7788410 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:32.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.293+0000 7fbb0e33db80 1 --2- 172.21.15.78:0/538236238 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c5e77396b0 0x55c5e775d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:32.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.293+0000 7fbb0e33db80 1 -- 172.21.15.78:0/538236238 >> 172.21.15.78:0/538236238 conn(0x55c5e7666d80 msgr2=0x55c5e772e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:32.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.293+0000 7fbada7fc700 0 client.5343 ms_handle_reset on v2:172.21.15.195:6833/2130671508 2023-10-14T11:59:32.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.293+0000 7fbb0e33db80 1 -- 172.21.15.78:0/538236238 shutdown_connections 2023-10-14T11:59:32.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.293+0000 7fbb0e33db80 1 -- 172.21.15.78:0/538236238 wait complete. 2023-10-14T11:59:32.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.293+0000 7fbb0e33db80 20 client.5343 trim_cache size 0 max 16384 2023-10-14T11:59:32.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.310+0000 7fbb0e33db80 1 Processor -- start 2023-10-14T11:59:32.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.310+0000 7fbb0e33db80 1 -- start start 2023-10-14T11:59:32.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.311+0000 7fbb0e33db80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c5e7733c90 0x55c5e776bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:32.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.311+0000 7fbb0e33db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c5e77396b0 0x55c5e77586f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:32.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.311+0000 7fbb0e33db80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55c5e7739a90 0x55c5e773a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:32.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.311+0000 7fbb0e33db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e750c8d0 con 0x55c5e77396b0 2023-10-14T11:59:32.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.311+0000 7fbb0e33db80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e7736ae0 con 0x55c5e7733c90 2023-10-14T11:59:32.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.311+0000 7fbb0e33db80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c5e7772a90 con 0x55c5e7739a90 2023-10-14T11:59:32.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.311+0000 7fbaf2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c5e77396b0 0x55c5e77586f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:32.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.311+0000 7fbaf2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c5e77396b0 0x55c5e77586f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46698/0 (socket says 172.21.15.78:46698) 2023-10-14T11:59:32.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.311+0000 7fbaf2ffd700 1 -- 172.21.15.78:0/495176850 learned_addr learned my addr 172.21.15.78:0/495176850 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:32.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.311+0000 7fbaf2ffd700 1 -- 172.21.15.78:0/495176850 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55c5e7739a90 msgr2=0x55c5e773a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:32.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.311+0000 7fbaf2ffd700 1 --2- 172.21.15.78:0/495176850 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55c5e7739a90 0x55c5e773a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:32.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.311+0000 7fbaf2ffd700 1 -- 172.21.15.78:0/495176850 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c5e7733c90 msgr2=0x55c5e776bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:32.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.311+0000 7fbaf2ffd700 1 --2- 172.21.15.78:0/495176850 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c5e7733c90 0x55c5e776bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:32.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.311+0000 7fbaf2ffd700 1 -- 172.21.15.78:0/495176850 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c5e749fc10 con 0x55c5e77396b0 2023-10-14T11:59:32.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.312+0000 7fbaf2ffd700 1 --2- 172.21.15.78:0/495176850 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c5e77396b0 0x55c5e77586f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fbae800c1b0 tx=0x7fbae400a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d58040f65877782 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:32.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.312+0000 7fbaf1ffb700 1 -- 172.21.15.78:0/495176850 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae402b070 con 0x55c5e77396b0 2023-10-14T11:59:32.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.312+0000 7fbaf1ffb700 1 -- 172.21.15.78:0/495176850 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbae4012800 con 0x55c5e77396b0 2023-10-14T11:59:32.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.312+0000 7fbaf1ffb700 1 -- 172.21.15.78:0/495176850 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae4016040 con 0x55c5e77396b0 2023-10-14T11:59:32.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.312+0000 7fbb0e33db80 1 -- 172.21.15.78:0/495176850 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c5e77396b0 msgr2=0x55c5e77586f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:32.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.312+0000 7fbb0e33db80 1 --2- 172.21.15.78:0/495176850 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c5e77396b0 0x55c5e77586f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fbae800c1b0 tx=0x7fbae400a000 comp rx=0 tx=0).stop 2023-10-14T11:59:32.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.312+0000 7fbb0e33db80 1 -- 172.21.15.78:0/495176850 shutdown_connections 2023-10-14T11:59:32.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.312+0000 7fbb0e33db80 1 --2- 172.21.15.78:0/495176850 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55c5e7739a90 0x55c5e773a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:32.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.312+0000 7fbb0e33db80 1 --2- 172.21.15.78:0/495176850 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c5e7733c90 0x55c5e776bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:32.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.312+0000 7fbb0e33db80 1 --2- 172.21.15.78:0/495176850 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c5e77396b0 0x55c5e77586f0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:32.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.312+0000 7fbb0e33db80 1 -- 172.21.15.78:0/495176850 >> 172.21.15.78:0/495176850 conn(0x55c5e7666d80 msgr2=0x55c5e7737fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:32.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.312+0000 7fbb0e33db80 1 -- 172.21.15.78:0/495176850 shutdown_connections 2023-10-14T11:59:32.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.312+0000 7fbb0e33db80 1 -- 172.21.15.78:0/495176850 wait complete. 2023-10-14T11:59:32.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.313+0000 7fbb0e33db80 1 -- start start 2023-10-14T11:59:32.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.313+0000 7fbb0e33db80 1 Processor -- start 2023-10-14T11:59:32.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.314+0000 7fbb0e33db80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55c5e7733c90 0x55c5e776d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:32.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.314+0000 7fbb0e33db80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c5e77396b0 0x55c5e77dc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:32.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.314+0000 7fbb0e33db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c5e7739a90 0x55c5e77d9570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:32.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.314+0000 7fbb0e33db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e7779930 con 0x55c5e7739a90 2023-10-14T11:59:32.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.314+0000 7fbb0e33db80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e77dd1e0 con 0x55c5e77396b0 2023-10-14T11:59:32.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.314+0000 7fbb0e33db80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c5e7755dc0 con 0x55c5e7733c90 2023-10-14T11:59:32.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.314+0000 7fbb0e33db80 10 client.0 _ll_register_callbacks cb 0x7fbaec058940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T11:59:32.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.314+0000 7fbafbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c5e7739a90 0x55c5e77d9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:32.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.314+0000 7fbafbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c5e7739a90 0x55c5e77d9570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46706/0 (socket says 172.21.15.78:46706) 2023-10-14T11:59:32.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.314+0000 7fbafbfff700 1 -- 172.21.15.78:0/3766986337 learned_addr learned my addr 172.21.15.78:0/3766986337 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:32.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.314+0000 7fbaf2ffd700 1 --2- 172.21.15.78:0/3766986337 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c5e77396b0 0x55c5e77dc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:32.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.314+0000 7fbafb7fe700 1 --2- 172.21.15.78:0/3766986337 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55c5e7733c90 0x55c5e776d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:32.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.314+0000 7fbafbfff700 1 -- 172.21.15.78:0/3766986337 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55c5e7733c90 msgr2=0x55c5e776d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:32.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.314+0000 7fbafbfff700 1 --2- 172.21.15.78:0/3766986337 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55c5e7733c90 0x55c5e776d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:32.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.314+0000 7fbafbfff700 1 -- 172.21.15.78:0/3766986337 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c5e77396b0 msgr2=0x55c5e77dc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:32.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.314+0000 7fbafbfff700 1 --2- 172.21.15.78:0/3766986337 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c5e77396b0 0x55c5e77dc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:32.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.314+0000 7fbafbfff700 1 -- 172.21.15.78:0/3766986337 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae80035b0 con 0x55c5e7739a90 2023-10-14T11:59:32.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.314+0000 7fbaf2ffd700 1 --2- 172.21.15.78:0/3766986337 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c5e77396b0 0x55c5e77dc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:59:32.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.314+0000 7fbafb7fe700 1 --2- 172.21.15.78:0/3766986337 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55c5e7733c90 0x55c5e776d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:59:32.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.314+0000 7fbafbfff700 1 --2- 172.21.15.78:0/3766986337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c5e7739a90 0x55c5e77d9570 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4001f50 tx=0x7fbaf4000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c04af015ee87bc72 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:32.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.314+0000 7fbada7fc700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-14T11:59:32.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.314+0000 7fbada7fc700 1 -- 172.21.15.78:0/3766986337 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf4014890 con 0x55c5e7739a90 2023-10-14T11:59:32.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.314+0000 7fbada7fc700 1 -- 172.21.15.78:0/3766986337 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7fbaf4002010 con 0x55c5e7739a90 2023-10-14T11:59:32.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.314+0000 7fbb0e33db80 1 -- 172.21.15.78:0/3766986337 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbaf4007b50 con 0x55c5e7739a90 2023-10-14T11:59:32.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.315+0000 7fbada7fc700 1 -- 172.21.15.78:0/3766986337 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbaf401fd00 con 0x55c5e7739a90 2023-10-14T11:59:32.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.315+0000 7fbb0e33db80 10 client.5349 fetch_fsmap learned FSMap version 17 2023-10-14T11:59:32.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.315+0000 7fbb0e33db80 1 -- 172.21.15.78:0/3766986337 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55c5e74dd200 con 0x55c5e7739a90 2023-10-14T11:59:32.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.315+0000 7fbada7fc700 1 -- 172.21.15.78:0/3766986337 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf4007b50 con 0x55c5e7739a90 2023-10-14T11:59:32.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.315+0000 7fbada7fc700 1 -- 172.21.15.78:0/3766986337 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 17) v1 ==== 3299+0+0 (secure 0 0 0) 0x7fbaf40073a0 con 0x55c5e7739a90 2023-10-14T11:59:32.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.315+0000 7fbb0e33db80 10 client.5349 fetch_fsmap finished waiting for FSMap version 17 2023-10-14T11:59:32.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.315+0000 7fbb0e33db80 10 client.5349 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-14T11:59:32.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.315+0000 7fbb0e33db80 20 client.5349 populate_metadata read hostname 'smithi078' 2023-10-14T11:59:32.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.315+0000 7fbb0e33db80 1 --2- 172.21.15.78:0/3766986337 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x55c5e77dca50 0x55c5e77d9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:32.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.315+0000 7fbb0e33db80 4 client.5349 mds_command: new command op to 5031 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-14T11:59:32.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.315+0000 7fbb0e33db80 1 -- 172.21.15.78:0/3766986337 --> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c5e7684b20 con 0x55c5e77dca50 2023-10-14T11:59:32.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.315+0000 7fbafb7fe700 1 --2- 172.21.15.78:0/3766986337 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x55c5e77dca50 0x55c5e77d9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:32.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.315+0000 7fbada7fc700 10 client.5349 ms_handle_connect on v2:172.21.15.195:6833/2130671508 2023-10-14T11:59:32.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.315+0000 7fbafb7fe700 1 --2- 172.21.15.78:0/3766986337 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x55c5e77dca50 0x55c5e77d9ab0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=912283031d06c848 server_cookie=78f9c19f46485af1 in_seq=0 out_seq=0 2023-10-14T11:59:32.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.316+0000 7fbada7fc700 1 -- 172.21.15.78:0/3766986337 <== mds.0 v2:172.21.15.195:6833/2130671508 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x55c5e7684b20 con 0x55c5e77dca50 2023-10-14T11:59:32.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.316+0000 7fbada7fc700 10 client.5349 handle_command_reply: tid=0 2023-10-14T11:59:32.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.316+0000 7fbb0e33db80 1 client.5349 shutdown 2023-10-14T11:59:32.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.316+0000 7fbb0e33db80 1 -- 172.21.15.78:0/3766986337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c5e7739a90 msgr2=0x55c5e77d9570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:32.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.316+0000 7fbb0e33db80 1 --2- 172.21.15.78:0/3766986337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c5e7739a90 0x55c5e77d9570 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4001f50 tx=0x7fbaf4000f40 comp rx=0 tx=0).stop 2023-10-14T11:59:32.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.316+0000 7fbb0e33db80 1 -- 172.21.15.78:0/3766986337 shutdown_connections 2023-10-14T11:59:32.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.316+0000 7fbb0e33db80 1 --2- 172.21.15.78:0/3766986337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c5e7739a90 0x55c5e77d9570 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:32.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.316+0000 7fbb0e33db80 1 --2- 172.21.15.78:0/3766986337 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55c5e7733c90 0x55c5e776d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:32.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.316+0000 7fbb0e33db80 1 --2- 172.21.15.78:0/3766986337 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x55c5e77dca50 0x55c5e77d9ab0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:32.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.316+0000 7fbb0e33db80 1 --2- 172.21.15.78:0/3766986337 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c5e77396b0 0x55c5e77dc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:32.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.316+0000 7fbada7fc700 0 client.5349 ms_handle_reset on v2:172.21.15.195:6833/2130671508 2023-10-14T11:59:32.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.316+0000 7fbb0e33db80 1 -- 172.21.15.78:0/3766986337 >> 172.21.15.78:0/3766986337 conn(0x55c5e7666d80 msgr2=0x55c5e773a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:32.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.316+0000 7fbb0e33db80 1 -- 172.21.15.78:0/3766986337 shutdown_connections 2023-10-14T11:59:32.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.316+0000 7fbb0e33db80 1 -- 172.21.15.78:0/3766986337 wait complete. 2023-10-14T11:59:32.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.316+0000 7fbb0e33db80 20 client.5349 trim_cache size 0 max 16384 2023-10-14T11:59:32.345 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-14T11:59:32.345 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "4c5dc134-3092-406a-a1e1-13e6d9e926e8", 2023-10-14T11:59:32.345 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2023-10-14T11:59:32.345 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 5031, 2023-10-14T11:59:32.346 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2023-10-14T11:59:32.346 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2023-10-14T11:59:32.346 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "testcephfs1", 2023-10-14T11:59:32.346 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 30.131894874, 2023-10-14T11:59:32.347 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 17, 2023-10-14T11:59:32.347 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 32, 2023-10-14T11:59:32.347 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 32, 2023-10-14T11:59:32.347 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 38.022717829000001 2023-10-14T11:59:32.347 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-14T11:59:32.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.316+0000 7fbb02153700 1 -- 172.21.15.78:0/1890921233 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fbadc06af00 msgr2=0x7fbadc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:32.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.316+0000 7fbb02153700 1 --2- 172.21.15.78:0/1890921233 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fbadc06af00 0x7fbadc06d3c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbafc154fd0 tx=0x7fbae4011000 comp rx=0 tx=0).stop 2023-10-14T11:59:32.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.316+0000 7fbb02153700 1 -- 172.21.15.78:0/1890921233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbae8004850 msgr2=0x7fbafc1552e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:32.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.316+0000 7fbb02153700 1 --2- 172.21.15.78:0/1890921233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbae8004850 0x7fbafc1552e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fbaec010810 tx=0x7fbaec010bd0 comp rx=0 tx=0).stop 2023-10-14T11:59:32.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.316+0000 7fbb02153700 1 -- 172.21.15.78:0/1890921233 shutdown_connections 2023-10-14T11:59:32.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.316+0000 7fbb02153700 1 --2- 172.21.15.78:0/1890921233 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbafc1edf50 0x7fbafc1ea420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:32.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.317+0000 7fbb02153700 1 --2- 172.21.15.78:0/1890921233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbae8004850 0x7fbafc1552e0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:32.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.317+0000 7fbb02153700 1 --2- 172.21.15.78:0/1890921233 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fbadc06af00 0x7fbadc06d3c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:32.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.317+0000 7fbb02153700 1 --2- 172.21.15.78:0/1890921233 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbafc06aab0 0x7fbafc1eda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:32.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.317+0000 7fbb02153700 1 -- 172.21.15.78:0/1890921233 >> 172.21.15.78:0/1890921233 conn(0x7fbafc0c3a20 msgr2=0x7fbafc151fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:32.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.317+0000 7fbb02153700 1 -- 172.21.15.78:0/1890921233 shutdown_connections 2023-10-14T11:59:32.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:32.317+0000 7fbb02153700 1 -- 172.21.15.78:0/1890921233 wait complete. 2023-10-14T11:59:32.351 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 17, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-14T11:59:01.562764+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 17, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "testcephfs1", "in": [ 0 ], "info": { "gid_5031": { "addr": "172.21.15.195:6835/2130671508", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6833", "nonce": 2130671508, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2130671508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5031, "incarnation": 13, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2023-10-14T11:59:04.592935+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5031 } } } ], "standbys": [ { "addr": "172.21.15.195:6834/1492486786", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6832", "nonce": 1492486786, "type": "v2" }, { "addr": "172.21.15.195:6834", "nonce": 1492486786, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4658, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6836/2991384297", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6834", "nonce": 2991384297, "type": "v2" }, { "addr": "172.21.15.78:6836", "nonce": 2991384297, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5016, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6837/3246640037", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6835", "nonce": 3246640037, "type": "v2" }, { "addr": "172.21.15.78:6837", "nonce": 3246640037, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5019, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-14T11:59:32.353 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 2, 'name': 'b', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2023-10-14T11:59:32.353 INFO:tasks.cephfs.test_failover:current = [{'gid': 5019, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6837/3246640037', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 3246640037}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 3246640037}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 4658, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.195:6834/1492486786', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6832', 'nonce': 1492486786}, {'type': 'v1', 'addr': '172.21.15.195:6834', 'nonce': 1492486786}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 5016, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6836/2991384297', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 2991384297}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 2991384297}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 5031, 'name': 'd', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.195:6835/2130671508', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6833', 'nonce': 2130671508}, {'type': 'v1', 'addr': '172.21.15.195:6835', 'nonce': 2130671508}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2023-10-14T11:59:32.354 DEBUG:tasks.cephfs.test_failover:2 != -1 2023-10-14T11:59:32.354 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2023-10-14T11:59:37.355 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T11:59:37.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.487+0000 7f02fcc96700 1 Processor -- start 2023-10-14T11:59:37.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.487+0000 7f02fcc96700 1 -- start start 2023-10-14T11:59:37.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.488+0000 7f02fcc96700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f02f806b910 0x7f02f80beb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:37.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.488+0000 7f02fcc96700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f02f80c2a60 0x7f02f80c2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:37.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.488+0000 7f02fcc96700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02f80c4910 con 0x7f02f81521b0 2023-10-14T11:59:37.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.488+0000 7f02fcc96700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02f80bcfa0 con 0x7f02f80c2a60 2023-10-14T11:59:37.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.488+0000 7f02fcc96700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02f8155980 con 0x7f02f806b910 2023-10-14T11:59:37.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.488+0000 7f02f6d9d700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f02f80c2a60 0x7f02f80c2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:37.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.488+0000 7f02f659c700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f02f81521b0 0x7f02f8155270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51586/0 (socket says 172.21.15.78:51586) 2023-10-14T11:59:37.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.488+0000 7f02f659c700 1 -- 172.21.15.78:0/215475503 learned_addr learned my addr 172.21.15.78:0/215475503 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:59:37.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.489+0000 7f02f5d9b700 1 --2- 172.21.15.78:0/215475503 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f02f806b910 0x7f02f80beb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:37.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.489+0000 7f02f6d9d700 1 -- 172.21.15.78:0/215475503 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f02f806b910 msgr2=0x7f02f80beb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:37.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.489+0000 7f02f6d9d700 1 --2- 172.21.15.78:0/215475503 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f02f806b910 0x7f02f80beb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:37.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.489+0000 7f02f6d9d700 1 -- 172.21.15.78:0/215475503 >> v1:172.21.15.78:6789/0 conn(0x7f02f81521b0 legacy=0x7f02f8155270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:37.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.489+0000 7f02f559a700 1 -- 172.21.15.78:0/215475503 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 153885459 0 0) 0x7f02f80c4910 con 0x7f02f81521b0 2023-10-14T11:59:37.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.489+0000 7f02f6d9d700 1 -- 172.21.15.78:0/215475503 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f8158aa0 con 0x7f02f80c2a60 2023-10-14T11:59:37.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.489+0000 7f02f6d9d700 1 --2- 172.21.15.78:0/215475503 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02f80c2a60 0x7f02f80c2e90 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f02f0009d90 tx=0x7f02f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4afa8a9c615b7a3f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:37.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.489+0000 7f02f559a700 1 -- 172.21.15.78:0/215475503 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f000e070 con 0x7f02f80c2a60 2023-10-14T11:59:37.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.489+0000 7f02f559a700 1 -- 172.21.15.78:0/215475503 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02f000bbd0 con 0x7f02f80c2a60 2023-10-14T11:59:37.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.489+0000 7f02f559a700 1 -- 172.21.15.78:0/215475503 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f0012430 con 0x7f02f80c2a60 2023-10-14T11:59:37.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.490+0000 7f02fcc96700 1 -- 172.21.15.78:0/215475503 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02f80c2a60 msgr2=0x7f02f80c2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:37.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.490+0000 7f02fcc96700 1 --2- 172.21.15.78:0/215475503 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02f80c2a60 0x7f02f80c2e90 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f02f0009d90 tx=0x7f02f000b330 comp rx=0 tx=0).stop 2023-10-14T11:59:37.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.490+0000 7f02fcc96700 1 -- 172.21.15.78:0/215475503 shutdown_connections 2023-10-14T11:59:37.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.490+0000 7f02fcc96700 1 --2- 172.21.15.78:0/215475503 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02f80c2a60 0x7f02f80c2e90 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:37.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.490+0000 7f02fcc96700 1 --2- 172.21.15.78:0/215475503 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f02f806b910 0x7f02f80beb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:37.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.490+0000 7f02fcc96700 1 -- 172.21.15.78:0/215475503 >> 172.21.15.78:0/215475503 conn(0x7f02f80c4b60 msgr2=0x7f02f80c4f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:37.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.490+0000 7f02fcc96700 1 -- 172.21.15.78:0/215475503 shutdown_connections 2023-10-14T11:59:37.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.490+0000 7f02fcc96700 1 -- 172.21.15.78:0/215475503 wait complete. 2023-10-14T11:59:37.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.491+0000 7f02fcc96700 1 Processor -- start 2023-10-14T11:59:37.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.492+0000 7f02fcc96700 1 -- start start 2023-10-14T11:59:37.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.492+0000 7f02fcc96700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02f806b910 0x7f02f81eaab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:37.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.492+0000 7f02fcc96700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f02f80c2a60 0x7f02f8142e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:37.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.492+0000 7f02fcc96700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f02f81521b0 0x7f02f80c6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:37.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.492+0000 7f02fcc96700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f80ad0d0 con 0x7f02f806b910 2023-10-14T11:59:37.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.492+0000 7f02fcc96700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f80c7440 con 0x7f02f80c2a60 2023-10-14T11:59:37.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.492+0000 7f02fcc96700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02f81eaff0 con 0x7f02f81521b0 2023-10-14T11:59:37.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.492+0000 7f02f659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02f806b910 0x7f02f81eaab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:37.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.493+0000 7f02f659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02f806b910 0x7f02f81eaab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46716/0 (socket says 172.21.15.78:46716) 2023-10-14T11:59:37.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.493+0000 7f02f659c700 1 -- 172.21.15.78:0/769317134 learned_addr learned my addr 172.21.15.78:0/769317134 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:37.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.493+0000 7f02f6d9d700 1 --2- 172.21.15.78:0/769317134 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f02f81521b0 0x7f02f80c6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:37.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.493+0000 7f02f5d9b700 1 --2- 172.21.15.78:0/769317134 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f02f80c2a60 0x7f02f8142e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:37.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.493+0000 7f02f659c700 1 -- 172.21.15.78:0/769317134 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f02f81521b0 msgr2=0x7f02f80c6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:37.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.493+0000 7f02f659c700 1 --2- 172.21.15.78:0/769317134 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f02f81521b0 0x7f02f80c6690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:37.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.493+0000 7f02f659c700 1 -- 172.21.15.78:0/769317134 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f02f80c2a60 msgr2=0x7f02f8142e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:37.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.493+0000 7f02f659c700 1 --2- 172.21.15.78:0/769317134 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f02f80c2a60 0x7f02f8142e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:37.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.493+0000 7f02f659c700 1 -- 172.21.15.78:0/769317134 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f0009a70 con 0x7f02f806b910 2023-10-14T11:59:37.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.493+0000 7f02f659c700 1 --2- 172.21.15.78:0/769317134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02f806b910 0x7f02f81eaab0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f02ec00c550 tx=0x7f02ec00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:37.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.493+0000 7f02e6ffd700 1 -- 172.21.15.78:0/769317134 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ec004c80 con 0x7f02f806b910 2023-10-14T11:59:37.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.493+0000 7f02e6ffd700 1 -- 172.21.15.78:0/769317134 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02ec004e00 con 0x7f02f806b910 2023-10-14T11:59:37.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.493+0000 7f02e6ffd700 1 -- 172.21.15.78:0/769317134 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ec007310 con 0x7f02f806b910 2023-10-14T11:59:37.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.493+0000 7f02fcc96700 1 -- 172.21.15.78:0/769317134 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02f81ed750 con 0x7f02f806b910 2023-10-14T11:59:37.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.493+0000 7f02fcc96700 1 -- 172.21.15.78:0/769317134 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02f806fdd0 con 0x7f02f806b910 2023-10-14T11:59:37.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.494+0000 7f02fcc96700 1 -- 172.21.15.78:0/769317134 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02f81ed970 con 0x7f02f806b910 2023-10-14T11:59:37.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.495+0000 7f02e6ffd700 1 -- 172.21.15.78:0/769317134 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f02ec007470 con 0x7f02f806b910 2023-10-14T11:59:37.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.495+0000 7f02e6ffd700 1 --2- 172.21.15.78:0/769317134 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f02d806af00 0x7f02d806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:37.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.495+0000 7f02f5d9b700 1 --2- 172.21.15.78:0/769317134 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f02d806af00 0x7f02d806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:37.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.495+0000 7f02e6ffd700 1 -- 172.21.15.78:0/769317134 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f02ec089280 con 0x7f02f806b910 2023-10-14T11:59:37.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.496+0000 7f02f5d9b700 1 --2- 172.21.15.78:0/769317134 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f02d806af00 0x7f02d806d3c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f02f814e320 tx=0x7f02e8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:37.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.500+0000 7f02e6ffd700 1 -- 172.21.15.78:0/769317134 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f02f81ed970 con 0x7f02f806b910 2023-10-14T11:59:37.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.732+0000 7f02fcc96700 1 -- 172.21.15.78:0/769317134 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02f81ed970 con 0x7f02f806b910 2023-10-14T11:59:37.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.732+0000 7f02e6ffd700 1 -- 172.21.15.78:0/769317134 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4924 (secure 0 0 0) 0x7f02f81ed970 con 0x7f02f806b910 2023-10-14T11:59:37.735 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:59:37.735 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4658,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6834/1492486786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1492486786},{"type":"v1","addr":"172.21.15.195:6834","nonce":1492486786}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5016,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/2991384297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2991384297},{"type":"v1","addr":"172.21.15.78:6836","nonce":2991384297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5019,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3246640037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3246640037},{"type":"v1","addr":"172.21.15.78:6837","nonce":3246640037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T11:59:01.562764+0000","modified":"2023-10-14T11:59:04.592935+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5031":{"gid":5031,"name":"d","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.195:6835/2130671508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6833","nonce":2130671508},{"type":"v1","addr":"172.21.15.195:6835","nonce":2130671508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"testcephfs1","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-14T11:59:37.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.735+0000 7f02fcc96700 1 -- 172.21.15.78:0/769317134 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f02d806af00 msgr2=0x7f02d806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:37.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.735+0000 7f02fcc96700 1 --2- 172.21.15.78:0/769317134 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f02d806af00 0x7f02d806d3c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f02f814e320 tx=0x7f02e8008040 comp rx=0 tx=0).stop 2023-10-14T11:59:37.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.735+0000 7f02fcc96700 1 -- 172.21.15.78:0/769317134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02f806b910 msgr2=0x7f02f81eaab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:37.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.735+0000 7f02fcc96700 1 --2- 172.21.15.78:0/769317134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02f806b910 0x7f02f81eaab0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f02ec00c550 tx=0x7f02ec00c910 comp rx=0 tx=0).stop 2023-10-14T11:59:37.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.735+0000 7f02fcc96700 1 -- 172.21.15.78:0/769317134 shutdown_connections 2023-10-14T11:59:37.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.735+0000 7f02fcc96700 1 --2- 172.21.15.78:0/769317134 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f02f81521b0 0x7f02f80c6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:37.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.735+0000 7f02fcc96700 1 --2- 172.21.15.78:0/769317134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02f806b910 0x7f02f81eaab0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:37.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.735+0000 7f02fcc96700 1 --2- 172.21.15.78:0/769317134 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f02d806af00 0x7f02d806d3c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:37.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.735+0000 7f02fcc96700 1 --2- 172.21.15.78:0/769317134 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f02f80c2a60 0x7f02f8142e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:37.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.735+0000 7f02fcc96700 1 -- 172.21.15.78:0/769317134 >> 172.21.15.78:0/769317134 conn(0x7f02f80c4b60 msgr2=0x7f02f806bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:37.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.735+0000 7f02fcc96700 1 -- 172.21.15.78:0/769317134 shutdown_connections 2023-10-14T11:59:37.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:37.735+0000 7f02fcc96700 1 -- 172.21.15.78:0/769317134 wait complete. 2023-10-14T11:59:37.740 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 17 2023-10-14T11:59:38.151 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-14T11:59:01.562764+0000', 'modified': '2023-10-14T11:59:04.592935+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5031}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5031': {'gid': 5031, 'name': 'd', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.195:6835/2130671508', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6833', 'nonce': 2130671508}, {'type': 'v1', 'addr': '172.21.15.195:6835', 'nonce': 2130671508}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'testcephfs1', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-14T11:59:38.152 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-14T11:59:38.152 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-14T11:59:38.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.281+0000 7fe77a8d1700 1 Processor -- start 2023-10-14T11:59:38.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.281+0000 7fe77a8d1700 1 -- start start 2023-10-14T11:59:38.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.282+0000 7fe77a8d1700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe7740a70c0 0x7fe774154960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:38.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.282+0000 7fe77a8d1700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7741537b0 con 0x7fe774154ea0 2023-10-14T11:59:38.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.282+0000 7fe77a8d1700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7740c7c50 con 0x7fe7741533d0 2023-10-14T11:59:38.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.282+0000 7fe77a8d1700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7740c6fb0 con 0x7fe7740a70c0 2023-10-14T11:59:38.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.282+0000 7fe778e6e700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fe774154ea0 0x7fe77415d130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51602/0 (socket says 172.21.15.78:51602) 2023-10-14T11:59:38.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.282+0000 7fe778e6e700 1 -- 172.21.15.78:0/1714008874 learned_addr learned my addr 172.21.15.78:0/1714008874 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:59:38.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.282+0000 7fe7737fe700 1 --2- 172.21.15.78:0/1714008874 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe7740a70c0 0x7fe774154960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:38.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.282+0000 7fe772ffd700 1 -- 172.21.15.78:0/1714008874 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2323253037 0 0) 0x7fe7741537b0 con 0x7fe774154ea0 2023-10-14T11:59:38.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.282+0000 7fe772ffd700 1 -- 172.21.15.78:0/1714008874 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe75c0035b0 con 0x7fe774154ea0 2023-10-14T11:59:38.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.282+0000 7fe772ffd700 1 -- 172.21.15.78:0/1714008874 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 630015795 0 0) 0x7fe75c0035b0 con 0x7fe774154ea0 2023-10-14T11:59:38.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.282+0000 7fe772ffd700 1 -- 172.21.15.78:0/1714008874 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe7740a70c0 msgr2=0x7fe774154960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:38.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.282+0000 7fe772ffd700 1 --2- 172.21.15.78:0/1714008874 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe7740a70c0 0x7fe774154960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:38.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.282+0000 7fe772ffd700 1 -- 172.21.15.78:0/1714008874 >> v1:172.21.15.195:6789/0 conn(0x7fe7741533d0 legacy=0x7fe7740c8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:38.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.282+0000 7fe772ffd700 1 -- 172.21.15.78:0/1714008874 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe774153e80 con 0x7fe774154ea0 2023-10-14T11:59:38.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.282+0000 7fe772ffd700 1 -- 172.21.15.78:0/1714008874 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441799917 0 0) 0x7fe76c0019f0 con 0x7fe774154ea0 2023-10-14T11:59:38.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.282+0000 7fe772ffd700 1 -- 172.21.15.78:0/1714008874 >> v1:172.21.15.78:6789/0 conn(0x7fe774154ea0 legacy=0x7fe77415d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:38.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.282+0000 7fe772ffd700 1 --2- 172.21.15.78:0/1714008874 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe75c003a30 0x7fe75c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:38.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.282+0000 7fe772ffd700 1 --2- 172.21.15.78:0/1714008874 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe7740a70c0 0x7fe75c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:38.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.282+0000 7fe772ffd700 1 --2- 172.21.15.78:0/1714008874 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe75c004d90 0x7fe75c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:38.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.282+0000 7fe772ffd700 1 -- 172.21.15.78:0/1714008874 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7740c6fb0 con 0x7fe75c003a30 2023-10-14T11:59:38.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.282+0000 7fe772ffd700 1 -- 172.21.15.78:0/1714008874 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75c00bb70 con 0x7fe75c004d90 2023-10-14T11:59:38.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.282+0000 7fe772ffd700 1 -- 172.21.15.78:0/1714008874 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe75c00bd30 con 0x7fe7740a70c0 2023-10-14T11:59:38.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.282+0000 7fe773fff700 1 --2- 172.21.15.78:0/1714008874 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe75c003a30 0x7fe75c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:38.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.282+0000 7fe7737fe700 1 --2- 172.21.15.78:0/1714008874 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe7740a70c0 0x7fe75c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:38.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.283+0000 7fe778e6e700 1 --2- 172.21.15.78:0/1714008874 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe75c004d90 0x7fe75c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:38.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.283+0000 7fe773fff700 1 -- 172.21.15.78:0/1714008874 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe7740a70c0 msgr2=0x7fe75c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:38.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.283+0000 7fe773fff700 1 --2- 172.21.15.78:0/1714008874 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe7740a70c0 0x7fe75c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:38.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.283+0000 7fe773fff700 1 -- 172.21.15.78:0/1714008874 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe75c004d90 msgr2=0x7fe75c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:38.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.283+0000 7fe773fff700 1 --2- 172.21.15.78:0/1714008874 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe75c004d90 0x7fe75c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:38.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.283+0000 7fe773fff700 1 -- 172.21.15.78:0/1714008874 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75c0035b0 con 0x7fe75c003a30 2023-10-14T11:59:38.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.283+0000 7fe773fff700 1 --2- 172.21.15.78:0/1714008874 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe75c003a30 0x7fe75c003e50 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe7680017c0 tx=0x7fe75c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e92453e4fb8b93b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:38.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.283+0000 7fe772ffd700 1 -- 172.21.15.78:0/1714008874 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7680052d0 con 0x7fe75c003a30 2023-10-14T11:59:38.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.283+0000 7fe772ffd700 1 -- 172.21.15.78:0/1714008874 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe768005430 con 0x7fe75c003a30 2023-10-14T11:59:38.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.283+0000 7fe772ffd700 1 -- 172.21.15.78:0/1714008874 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7680038c0 con 0x7fe75c003a30 2023-10-14T11:59:38.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.283+0000 7fe77a8d1700 1 -- 172.21.15.78:0/1714008874 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe75c003a30 msgr2=0x7fe75c003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:38.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.283+0000 7fe77a8d1700 1 --2- 172.21.15.78:0/1714008874 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe75c003a30 0x7fe75c003e50 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe7680017c0 tx=0x7fe75c005880 comp rx=0 tx=0).stop 2023-10-14T11:59:38.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.283+0000 7fe778e6e700 1 -- 172.21.15.78:0/1714008874 reap_dead start 2023-10-14T11:59:38.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.283+0000 7fe77a8d1700 1 -- 172.21.15.78:0/1714008874 shutdown_connections 2023-10-14T11:59:38.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.283+0000 7fe77a8d1700 1 -- 172.21.15.78:0/1714008874 >> 172.21.15.78:0/1714008874 conn(0x7fe7740737d0 msgr2=0x7fe774073be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:38.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.283+0000 7fe77a8d1700 1 -- 172.21.15.78:0/1714008874 shutdown_connections 2023-10-14T11:59:38.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.283+0000 7fe77a8d1700 1 -- 172.21.15.78:0/1714008874 wait complete. 2023-10-14T11:59:38.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.285+0000 7fe77a8d1700 1 Processor -- start 2023-10-14T11:59:38.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.285+0000 7fe77a8d1700 1 -- start start 2023-10-14T11:59:38.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.285+0000 7fe77a8d1700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe75c003a30 0x7fe7740ad7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:38.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.285+0000 7fe77a8d1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe7741ed490 0x7fe77414bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:38.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.285+0000 7fe77a8d1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe77406cc00 0x7fe77414c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:38.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.285+0000 7fe77a8d1700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7740c7f30 con 0x7fe77406cc00 2023-10-14T11:59:38.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.285+0000 7fe77a8d1700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7740c7a70 con 0x7fe7741ed490 2023-10-14T11:59:38.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.285+0000 7fe77a8d1700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7741eda00 con 0x7fe75c003a30 2023-10-14T11:59:38.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.286+0000 7fe778e6e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe77406cc00 0x7fe77414c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:38.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.286+0000 7fe778e6e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe77406cc00 0x7fe77414c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46736/0 (socket says 172.21.15.78:46736) 2023-10-14T11:59:38.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.286+0000 7fe778e6e700 1 -- 172.21.15.78:0/4091904686 learned_addr learned my addr 172.21.15.78:0/4091904686 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:38.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.286+0000 7fe773fff700 1 --2- 172.21.15.78:0/4091904686 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe75c003a30 0x7fe7740ad7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:38.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.286+0000 7fe7737fe700 1 --2- 172.21.15.78:0/4091904686 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe7741ed490 0x7fe77414bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:38.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.286+0000 7fe778e6e700 1 -- 172.21.15.78:0/4091904686 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe75c003a30 msgr2=0x7fe7740ad7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:38.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.286+0000 7fe778e6e700 1 --2- 172.21.15.78:0/4091904686 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe75c003a30 0x7fe7740ad7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:38.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.286+0000 7fe778e6e700 1 -- 172.21.15.78:0/4091904686 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe7741ed490 msgr2=0x7fe77414bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:38.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.286+0000 7fe778e6e700 1 --2- 172.21.15.78:0/4091904686 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe7741ed490 0x7fe77414bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:38.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.286+0000 7fe778e6e700 1 -- 172.21.15.78:0/4091904686 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7741edce0 con 0x7fe77406cc00 2023-10-14T11:59:38.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.286+0000 7fe778e6e700 1 --2- 172.21.15.78:0/4091904686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe77406cc00 0x7fe77414c1e0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fe774156dc0 tx=0x7fe76c00e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:38.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.286+0000 7fe770ff9700 1 -- 172.21.15.78:0/4091904686 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76c017070 con 0x7fe77406cc00 2023-10-14T11:59:38.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.286+0000 7fe770ff9700 1 -- 172.21.15.78:0/4091904686 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe76c00ec90 con 0x7fe77406cc00 2023-10-14T11:59:38.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.286+0000 7fe77a8d1700 1 -- 172.21.15.78:0/4091904686 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7741edfc0 con 0x7fe77406cc00 2023-10-14T11:59:38.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.286+0000 7fe770ff9700 1 -- 172.21.15.78:0/4091904686 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76c00ee40 con 0x7fe77406cc00 2023-10-14T11:59:38.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.286+0000 7fe77a8d1700 1 -- 172.21.15.78:0/4091904686 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7740aace0 con 0x7fe77406cc00 2023-10-14T11:59:38.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.288+0000 7fe770ff9700 1 -- 172.21.15.78:0/4091904686 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe76c013420 con 0x7fe77406cc00 2023-10-14T11:59:38.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.288+0000 7fe770ff9700 1 --2- 172.21.15.78:0/4091904686 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe76006af00 0x7fe76006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:38.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.288+0000 7fe770ff9700 1 -- 172.21.15.78:0/4091904686 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7fe76c0891c0 con 0x7fe77406cc00 2023-10-14T11:59:38.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.288+0000 7fe773fff700 1 --2- 172.21.15.78:0/4091904686 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe76006af00 0x7fe76006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:38.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.289+0000 7fe773fff700 1 --2- 172.21.15.78:0/4091904686 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe76006af00 0x7fe76006d3c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe768000c00 tx=0x7fe768009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:38.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.290+0000 7fe786abbb80 1 Processor -- start 2023-10-14T11:59:38.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.290+0000 7fe786abbb80 1 -- start start 2023-10-14T11:59:38.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.291+0000 7fe786abbb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d11e166b0 0x563d11e54620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:38.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.291+0000 7fe786abbb80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563d11e16a90 0x563d11e54b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:38.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.291+0000 7fe786abbb80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x563d11e10c90 0x563d11e348a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:38.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.291+0000 7fe786abbb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d11d8ac20 con 0x563d11e166b0 2023-10-14T11:59:38.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.291+0000 7fe786abbb80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d11d6fc10 con 0x563d11e16a90 2023-10-14T11:59:38.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.291+0000 7fe786abbb80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x563d11d6d530 con 0x563d11e10c90 2023-10-14T11:59:38.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.291+0000 7fe773fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d11e166b0 0x563d11e54620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:38.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.291+0000 7fe773fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d11e166b0 0x563d11e54620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46740/0 (socket says 172.21.15.78:46740) 2023-10-14T11:59:38.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.291+0000 7fe773fff700 1 -- 172.21.15.78:0/1190653759 learned_addr learned my addr 172.21.15.78:0/1190653759 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:38.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.291+0000 7fe773fff700 1 -- 172.21.15.78:0/1190653759 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x563d11e10c90 msgr2=0x563d11e348a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:38.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.291+0000 7fe773fff700 1 --2- 172.21.15.78:0/1190653759 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x563d11e10c90 0x563d11e348a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:38.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.291+0000 7fe773fff700 1 -- 172.21.15.78:0/1190653759 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563d11e16a90 msgr2=0x563d11e54b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:38.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.291+0000 7fe773fff700 1 --2- 172.21.15.78:0/1190653759 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563d11e16a90 0x563d11e54b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:38.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.291+0000 7fe773fff700 1 -- 172.21.15.78:0/1190653759 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d11d263a0 con 0x563d11e166b0 2023-10-14T11:59:38.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.291+0000 7fe773fff700 1 --2- 172.21.15.78:0/1190653759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d11e166b0 0x563d11e54620 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe768001db0 tx=0x7fe768011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef53a0f4f9907fa server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:38.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.291+0000 7fe759ffb700 1 -- 172.21.15.78:0/1190653759 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe768003880 con 0x563d11e166b0 2023-10-14T11:59:38.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.291+0000 7fe759ffb700 1 -- 172.21.15.78:0/1190653759 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe768003a00 con 0x563d11e166b0 2023-10-14T11:59:38.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.291+0000 7fe759ffb700 1 -- 172.21.15.78:0/1190653759 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe768003db0 con 0x563d11e166b0 2023-10-14T11:59:38.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.292+0000 7fe786abbb80 1 -- 172.21.15.78:0/1190653759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d11e166b0 msgr2=0x563d11e54620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:38.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.292+0000 7fe786abbb80 1 --2- 172.21.15.78:0/1190653759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d11e166b0 0x563d11e54620 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe768001db0 tx=0x7fe768011000 comp rx=0 tx=0).stop 2023-10-14T11:59:38.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.292+0000 7fe786abbb80 1 -- 172.21.15.78:0/1190653759 shutdown_connections 2023-10-14T11:59:38.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.292+0000 7fe786abbb80 1 --2- 172.21.15.78:0/1190653759 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x563d11e10c90 0x563d11e348a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:38.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.292+0000 7fe786abbb80 1 --2- 172.21.15.78:0/1190653759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d11e166b0 0x563d11e54620 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:38.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.292+0000 7fe786abbb80 1 --2- 172.21.15.78:0/1190653759 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563d11e16a90 0x563d11e54b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:38.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.292+0000 7fe786abbb80 1 -- 172.21.15.78:0/1190653759 >> 172.21.15.78:0/1190653759 conn(0x563d11d43d80 msgr2=0x563d11e143a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:38.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.292+0000 7fe786abbb80 1 -- 172.21.15.78:0/1190653759 shutdown_connections 2023-10-14T11:59:38.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.292+0000 7fe786abbb80 1 -- 172.21.15.78:0/1190653759 wait complete. 2023-10-14T11:59:38.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.293+0000 7fe786abbb80 1 -- start start 2023-10-14T11:59:38.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.293+0000 7fe786abbb80 1 Processor -- start 2023-10-14T11:59:38.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.294+0000 7fe786abbb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d11e10c90 0x563d11e331e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:38.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.294+0000 7fe786abbb80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x563d11e166b0 0x563d11e62230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:38.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.294+0000 7fe786abbb80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563d11e16a90 0x563d11e3ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:38.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.294+0000 7fe786abbb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d11d68e20 con 0x563d11e10c90 2023-10-14T11:59:38.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.294+0000 7fe786abbb80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d11d056c0 con 0x563d11e16a90 2023-10-14T11:59:38.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.294+0000 7fe786abbb80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x563d11d05360 con 0x563d11e166b0 2023-10-14T11:59:38.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.294+0000 7fe786abbb80 10 client.0 _ll_register_callbacks cb 0x563d11b627b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T11:59:38.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.294+0000 7fe773fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d11e10c90 0x563d11e331e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:38.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.294+0000 7fe773fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d11e10c90 0x563d11e331e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46746/0 (socket says 172.21.15.78:46746) 2023-10-14T11:59:38.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.294+0000 7fe773fff700 1 -- 172.21.15.78:0/1938501686 learned_addr learned my addr 172.21.15.78:0/1938501686 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:38.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.294+0000 7fe7737fe700 1 --2- 172.21.15.78:0/1938501686 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x563d11e166b0 0x563d11e62230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:38.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.294+0000 7fe773fff700 1 -- 172.21.15.78:0/1938501686 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x563d11e166b0 msgr2=0x563d11e62230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:38.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.294+0000 7fe778e6e700 1 --2- 172.21.15.78:0/1938501686 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563d11e16a90 0x563d11e3ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:38.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.294+0000 7fe773fff700 1 --2- 172.21.15.78:0/1938501686 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x563d11e166b0 0x563d11e62230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:38.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.294+0000 7fe773fff700 1 -- 172.21.15.78:0/1938501686 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563d11e16a90 msgr2=0x563d11e3ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:38.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.294+0000 7fe773fff700 1 --2- 172.21.15.78:0/1938501686 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563d11e16a90 0x563d11e3ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:38.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.294+0000 7fe773fff700 1 -- 172.21.15.78:0/1938501686 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe75c0035b0 con 0x563d11e10c90 2023-10-14T11:59:38.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.295+0000 7fe773fff700 1 --2- 172.21.15.78:0/1938501686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d11e10c90 0x563d11e331e0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fe768005040 tx=0x7fe76800f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22ad6e42458e4bf2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:38.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.295+0000 7fe752ffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-14T11:59:38.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.295+0000 7fe752ffd700 1 -- 172.21.15.78:0/1938501686 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76803a070 con 0x563d11e10c90 2023-10-14T11:59:38.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.295+0000 7fe752ffd700 1 -- 172.21.15.78:0/1938501686 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7fe76803fae0 con 0x563d11e10c90 2023-10-14T11:59:38.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.295+0000 7fe786abbb80 1 -- 172.21.15.78:0/1938501686 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563d11e095b0 con 0x563d11e10c90 2023-10-14T11:59:38.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.295+0000 7fe752ffd700 1 -- 172.21.15.78:0/1938501686 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe768003e80 con 0x563d11e10c90 2023-10-14T11:59:38.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.295+0000 7fe786abbb80 10 client.5370 fetch_fsmap learned FSMap version 17 2023-10-14T11:59:38.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.295+0000 7fe786abbb80 1 -- 172.21.15.78:0/1938501686 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x563d11b93d90 con 0x563d11e10c90 2023-10-14T11:59:38.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.295+0000 7fe752ffd700 1 -- 172.21.15.78:0/1938501686 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563d11e095b0 con 0x563d11e10c90 2023-10-14T11:59:38.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.295+0000 7fe752ffd700 1 -- 172.21.15.78:0/1938501686 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 17) v1 ==== 3299+0+0 (secure 0 0 0) 0x7fe76803fca0 con 0x563d11e10c90 2023-10-14T11:59:38.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.295+0000 7fe786abbb80 10 client.5370 fetch_fsmap finished waiting for FSMap version 17 2023-10-14T11:59:38.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.295+0000 7fe786abbb80 10 client.5370 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-14T11:59:38.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.295+0000 7fe786abbb80 20 client.5370 populate_metadata read hostname 'smithi078' 2023-10-14T11:59:38.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.295+0000 7fe786abbb80 1 --2- 172.21.15.78:0/1938501686 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x563d11e3b090 0x563d11e50590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:38.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.295+0000 7fe786abbb80 4 client.5370 mds_command: new command op to 5031 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-14T11:59:38.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.295+0000 7fe786abbb80 1 -- 172.21.15.78:0/1938501686 --> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563d11d61e00 con 0x563d11e3b090 2023-10-14T11:59:38.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.295+0000 7fe7737fe700 1 --2- 172.21.15.78:0/1938501686 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x563d11e3b090 0x563d11e50590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:38.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.296+0000 7fe752ffd700 10 client.5370 ms_handle_connect on v2:172.21.15.195:6833/2130671508 2023-10-14T11:59:38.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.296+0000 7fe7737fe700 1 --2- 172.21.15.78:0/1938501686 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x563d11e3b090 0x563d11e50590 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bab157d03735e076 server_cookie=2004e3fac213ba0d in_seq=0 out_seq=0 2023-10-14T11:59:38.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.296+0000 7fe752ffd700 1 -- 172.21.15.78:0/1938501686 <== mds.0 v2:172.21.15.195:6833/2130671508 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x563d11d61e00 con 0x563d11e3b090 2023-10-14T11:59:38.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.296+0000 7fe752ffd700 10 client.5370 handle_command_reply: tid=0 2023-10-14T11:59:38.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.297+0000 7fe786abbb80 1 client.5370 shutdown 2023-10-14T11:59:38.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.297+0000 7fe786abbb80 1 -- 172.21.15.78:0/1938501686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d11e10c90 msgr2=0x563d11e331e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:38.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.297+0000 7fe786abbb80 1 --2- 172.21.15.78:0/1938501686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d11e10c90 0x563d11e331e0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fe768005040 tx=0x7fe76800f000 comp rx=0 tx=0).stop 2023-10-14T11:59:38.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.297+0000 7fe786abbb80 1 -- 172.21.15.78:0/1938501686 shutdown_connections 2023-10-14T11:59:38.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.297+0000 7fe786abbb80 1 --2- 172.21.15.78:0/1938501686 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x563d11e3b090 0x563d11e50590 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:38.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.297+0000 7fe786abbb80 1 --2- 172.21.15.78:0/1938501686 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563d11e16a90 0x563d11e3ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:38.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.297+0000 7fe752ffd700 0 client.5370 ms_handle_reset on v2:172.21.15.195:6833/2130671508 2023-10-14T11:59:38.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.297+0000 7fe786abbb80 1 --2- 172.21.15.78:0/1938501686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d11e10c90 0x563d11e331e0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:38.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.297+0000 7fe786abbb80 1 --2- 172.21.15.78:0/1938501686 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x563d11e166b0 0x563d11e62230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:38.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.297+0000 7fe786abbb80 1 -- 172.21.15.78:0/1938501686 >> 172.21.15.78:0/1938501686 conn(0x563d11d43d80 msgr2=0x563d11e0b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:38.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.297+0000 7fe786abbb80 1 -- 172.21.15.78:0/1938501686 shutdown_connections 2023-10-14T11:59:38.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.297+0000 7fe786abbb80 1 -- 172.21.15.78:0/1938501686 wait complete. 2023-10-14T11:59:38.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.297+0000 7fe786abbb80 20 client.5370 trim_cache size 0 max 16384 2023-10-14T11:59:38.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.314+0000 7fe786abbb80 1 Processor -- start 2023-10-14T11:59:38.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.314+0000 7fe786abbb80 1 -- start start 2023-10-14T11:59:38.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.315+0000 7fe786abbb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d11e10c90 0x563d11e56680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:38.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.315+0000 7fe786abbb80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x563d11e166b0 0x563d11e65500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:38.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.315+0000 7fe786abbb80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563d11e16a90 0x563d11e65a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:38.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.315+0000 7fe786abbb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d11e136b0 con 0x563d11e10c90 2023-10-14T11:59:38.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.315+0000 7fe786abbb80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d11be98d0 con 0x563d11e16a90 2023-10-14T11:59:38.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.315+0000 7fe786abbb80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x563d11e2fed0 con 0x563d11e166b0 2023-10-14T11:59:38.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.315+0000 7fe773fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d11e10c90 0x563d11e56680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:38.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.315+0000 7fe773fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d11e10c90 0x563d11e56680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46754/0 (socket says 172.21.15.78:46754) 2023-10-14T11:59:38.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.315+0000 7fe773fff700 1 -- 172.21.15.78:0/1581423881 learned_addr learned my addr 172.21.15.78:0/1581423881 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:38.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.315+0000 7fe7737fe700 1 --2- 172.21.15.78:0/1581423881 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x563d11e166b0 0x563d11e65500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:38.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.315+0000 7fe778e6e700 1 --2- 172.21.15.78:0/1581423881 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563d11e16a90 0x563d11e65a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:38.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.315+0000 7fe773fff700 1 -- 172.21.15.78:0/1581423881 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x563d11e166b0 msgr2=0x563d11e65500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:38.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.315+0000 7fe773fff700 1 --2- 172.21.15.78:0/1581423881 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x563d11e166b0 0x563d11e65500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:38.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.315+0000 7fe773fff700 1 -- 172.21.15.78:0/1581423881 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563d11e16a90 msgr2=0x563d11e65a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:38.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.315+0000 7fe773fff700 1 --2- 172.21.15.78:0/1581423881 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563d11e16a90 0x563d11e65a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:38.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.315+0000 7fe773fff700 1 -- 172.21.15.78:0/1581423881 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d11b7cc10 con 0x563d11e10c90 2023-10-14T11:59:38.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.315+0000 7fe778e6e700 1 --2- 172.21.15.78:0/1581423881 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563d11e16a90 0x563d11e65a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:59:38.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.316+0000 7fe773fff700 1 --2- 172.21.15.78:0/1581423881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d11e10c90 0x563d11e56680 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fe7680051a0 tx=0x7fe76802c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=788cfdd9fea9b2ea server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:38.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.316+0000 7fe75a7fc700 1 -- 172.21.15.78:0/1581423881 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe768048470 con 0x563d11e10c90 2023-10-14T11:59:38.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.316+0000 7fe75a7fc700 1 -- 172.21.15.78:0/1581423881 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7680485d0 con 0x563d11e10c90 2023-10-14T11:59:38.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.316+0000 7fe75a7fc700 1 -- 172.21.15.78:0/1581423881 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe768048940 con 0x563d11e10c90 2023-10-14T11:59:38.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.316+0000 7fe786abbb80 1 -- 172.21.15.78:0/1581423881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d11e10c90 msgr2=0x563d11e56680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:38.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.316+0000 7fe786abbb80 1 --2- 172.21.15.78:0/1581423881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d11e10c90 0x563d11e56680 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fe7680051a0 tx=0x7fe76802c000 comp rx=0 tx=0).stop 2023-10-14T11:59:38.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.316+0000 7fe786abbb80 1 -- 172.21.15.78:0/1581423881 shutdown_connections 2023-10-14T11:59:38.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.316+0000 7fe786abbb80 1 --2- 172.21.15.78:0/1581423881 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563d11e16a90 0x563d11e65a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:38.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.316+0000 7fe786abbb80 1 --2- 172.21.15.78:0/1581423881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d11e10c90 0x563d11e56680 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:38.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.316+0000 7fe786abbb80 1 --2- 172.21.15.78:0/1581423881 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x563d11e166b0 0x563d11e65500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:38.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.316+0000 7fe786abbb80 1 -- 172.21.15.78:0/1581423881 >> 172.21.15.78:0/1581423881 conn(0x563d11d43d80 msgr2=0x563d11e74790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:38.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.316+0000 7fe786abbb80 1 -- 172.21.15.78:0/1581423881 shutdown_connections 2023-10-14T11:59:38.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.316+0000 7fe786abbb80 1 -- 172.21.15.78:0/1581423881 wait complete. 2023-10-14T11:59:38.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.317+0000 7fe786abbb80 1 -- start start 2023-10-14T11:59:38.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.317+0000 7fe786abbb80 1 Processor -- start 2023-10-14T11:59:38.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.318+0000 7fe786abbb80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x563d11e10c90 0x563d11e65870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:38.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.318+0000 7fe786abbb80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563d11e166b0 0x563d11ebc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:38.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.318+0000 7fe786abbb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d11e16a90 0x563d11eb60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:38.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.318+0000 7fe786abbb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d11e17a60 con 0x563d11e16a90 2023-10-14T11:59:38.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.318+0000 7fe786abbb80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d11e15d60 con 0x563d11e166b0 2023-10-14T11:59:38.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.318+0000 7fe786abbb80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x563d11ebd020 con 0x563d11e10c90 2023-10-14T11:59:38.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.318+0000 7fe786abbb80 10 client.0 _ll_register_callbacks cb 0x7fe768001690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T11:59:38.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.318+0000 7fe778e6e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d11e16a90 0x563d11eb60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:38.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.318+0000 7fe778e6e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d11e16a90 0x563d11eb60e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46764/0 (socket says 172.21.15.78:46764) 2023-10-14T11:59:38.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.318+0000 7fe778e6e700 1 -- 172.21.15.78:0/707546447 learned_addr learned my addr 172.21.15.78:0/707546447 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:38.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.318+0000 7fe773fff700 1 --2- 172.21.15.78:0/707546447 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x563d11e10c90 0x563d11e65870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:38.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.318+0000 7fe7737fe700 1 --2- 172.21.15.78:0/707546447 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563d11e166b0 0x563d11ebc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:38.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.319+0000 7fe778e6e700 1 -- 172.21.15.78:0/707546447 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x563d11e10c90 msgr2=0x563d11e65870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:38.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.319+0000 7fe778e6e700 1 --2- 172.21.15.78:0/707546447 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x563d11e10c90 0x563d11e65870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:38.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.319+0000 7fe778e6e700 1 -- 172.21.15.78:0/707546447 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563d11e166b0 msgr2=0x563d11ebc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:38.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.319+0000 7fe778e6e700 1 --2- 172.21.15.78:0/707546447 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563d11e166b0 0x563d11ebc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:38.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.319+0000 7fe778e6e700 1 -- 172.21.15.78:0/707546447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563d11b93d90 con 0x563d11e16a90 2023-10-14T11:59:38.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.319+0000 7fe773fff700 1 --2- 172.21.15.78:0/707546447 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x563d11e10c90 0x563d11e65870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:59:38.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.319+0000 7fe778e6e700 1 --2- 172.21.15.78:0/707546447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d11e16a90 0x563d11eb60e0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fe76c087700 tx=0x7fe76c0586a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1108ba7f63f57a7b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:38.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.319+0000 7fe752ffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-14T11:59:38.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.319+0000 7fe752ffd700 1 -- 172.21.15.78:0/707546447 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76c058400 con 0x563d11e16a90 2023-10-14T11:59:38.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.319+0000 7fe752ffd700 1 -- 172.21.15.78:0/707546447 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7fe76c0602a0 con 0x563d11e16a90 2023-10-14T11:59:38.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.319+0000 7fe786abbb80 1 -- 172.21.15.78:0/707546447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe768003a00 con 0x563d11e16a90 2023-10-14T11:59:38.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.319+0000 7fe752ffd700 1 -- 172.21.15.78:0/707546447 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe76c057490 con 0x563d11e16a90 2023-10-14T11:59:38.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.319+0000 7fe786abbb80 10 client.5376 fetch_fsmap learned FSMap version 17 2023-10-14T11:59:38.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.319+0000 7fe786abbb80 1 -- 172.21.15.78:0/707546447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x563d11bba200 con 0x563d11e16a90 2023-10-14T11:59:38.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.319+0000 7fe752ffd700 1 -- 172.21.15.78:0/707546447 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe768003a00 con 0x563d11e16a90 2023-10-14T11:59:38.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.319+0000 7fe752ffd700 1 -- 172.21.15.78:0/707546447 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 17) v1 ==== 3299+0+0 (secure 0 0 0) 0x7fe76c05fe10 con 0x563d11e16a90 2023-10-14T11:59:38.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.319+0000 7fe786abbb80 10 client.5376 fetch_fsmap finished waiting for FSMap version 17 2023-10-14T11:59:38.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.319+0000 7fe786abbb80 10 client.5376 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-14T11:59:38.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.319+0000 7fe786abbb80 20 client.5376 populate_metadata read hostname 'smithi078' 2023-10-14T11:59:38.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.319+0000 7fe786abbb80 1 --2- 172.21.15.78:0/707546447 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x563d11eb6740 0x563d11e33dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:38.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.319+0000 7fe786abbb80 4 client.5376 mds_command: new command op to 5031 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-14T11:59:38.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.319+0000 7fe786abbb80 1 -- 172.21.15.78:0/707546447 --> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] -- command(tid 0: {"prefix": "status"}) v1 -- 0x563d11d61b20 con 0x563d11eb6740 2023-10-14T11:59:38.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.320+0000 7fe773fff700 1 --2- 172.21.15.78:0/707546447 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x563d11eb6740 0x563d11e33dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:38.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.320+0000 7fe752ffd700 10 client.5376 ms_handle_connect on v2:172.21.15.195:6833/2130671508 2023-10-14T11:59:38.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.320+0000 7fe773fff700 1 --2- 172.21.15.78:0/707546447 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x563d11eb6740 0x563d11e33dd0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a7fc322a9c9be5d server_cookie=4e4c5764b2cb0776 in_seq=0 out_seq=0 2023-10-14T11:59:38.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.320+0000 7fe752ffd700 1 -- 172.21.15.78:0/707546447 <== mds.0 v2:172.21.15.195:6833/2130671508 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+336 (crc 0 0 0) 0x563d11d61b20 con 0x563d11eb6740 2023-10-14T11:59:38.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.320+0000 7fe752ffd700 10 client.5376 handle_command_reply: tid=0 2023-10-14T11:59:38.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.320+0000 7fe786abbb80 1 client.5376 shutdown 2023-10-14T11:59:38.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.320+0000 7fe786abbb80 1 -- 172.21.15.78:0/707546447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d11e16a90 msgr2=0x563d11eb60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:38.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.320+0000 7fe786abbb80 1 --2- 172.21.15.78:0/707546447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d11e16a90 0x563d11eb60e0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fe76c087700 tx=0x7fe76c0586a0 comp rx=0 tx=0).stop 2023-10-14T11:59:38.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.320+0000 7fe786abbb80 1 -- 172.21.15.78:0/707546447 shutdown_connections 2023-10-14T11:59:38.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.320+0000 7fe786abbb80 1 --2- 172.21.15.78:0/707546447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d11e16a90 0x563d11eb60e0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:38.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.320+0000 7fe786abbb80 1 --2- 172.21.15.78:0/707546447 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x563d11e10c90 0x563d11e65870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:38.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.320+0000 7fe786abbb80 1 --2- 172.21.15.78:0/707546447 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x563d11eb6740 0x563d11e33dd0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:38.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.320+0000 7fe786abbb80 1 --2- 172.21.15.78:0/707546447 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563d11e166b0 0x563d11ebc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:38.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.320+0000 7fe752ffd700 0 client.5376 ms_handle_reset on v2:172.21.15.195:6833/2130671508 2023-10-14T11:59:38.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.320+0000 7fe786abbb80 1 -- 172.21.15.78:0/707546447 >> 172.21.15.78:0/707546447 conn(0x563d11d43d80 msgr2=0x563d11e74790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:38.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.320+0000 7fe786abbb80 1 -- 172.21.15.78:0/707546447 shutdown_connections 2023-10-14T11:59:38.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.320+0000 7fe786abbb80 1 -- 172.21.15.78:0/707546447 wait complete. 2023-10-14T11:59:38.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.320+0000 7fe786abbb80 20 client.5376 trim_cache size 0 max 16384 2023-10-14T11:59:38.358 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-14T11:59:38.358 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "4c5dc134-3092-406a-a1e1-13e6d9e926e8", 2023-10-14T11:59:38.358 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2023-10-14T11:59:38.359 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 5031, 2023-10-14T11:59:38.359 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2023-10-14T11:59:38.359 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2023-10-14T11:59:38.359 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "testcephfs1", 2023-10-14T11:59:38.359 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 36.136260776999997, 2023-10-14T11:59:38.360 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 17, 2023-10-14T11:59:38.360 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 32, 2023-10-14T11:59:38.360 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 32, 2023-10-14T11:59:38.360 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 44.027084512000002 2023-10-14T11:59:38.360 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-14T11:59:38.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.321+0000 7fe77a8d1700 1 -- 172.21.15.78:0/4091904686 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe76006af00 msgr2=0x7fe76006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:38.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.321+0000 7fe77a8d1700 1 --2- 172.21.15.78:0/4091904686 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe76006af00 0x7fe76006d3c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe768000c00 tx=0x7fe768009030 comp rx=0 tx=0).stop 2023-10-14T11:59:38.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.321+0000 7fe77a8d1700 1 -- 172.21.15.78:0/4091904686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe77406cc00 msgr2=0x7fe77414c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:38.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.321+0000 7fe77a8d1700 1 --2- 172.21.15.78:0/4091904686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe77406cc00 0x7fe77414c1e0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fe774156dc0 tx=0x7fe76c00e440 comp rx=0 tx=0).stop 2023-10-14T11:59:38.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.321+0000 7fe77a8d1700 1 -- 172.21.15.78:0/4091904686 shutdown_connections 2023-10-14T11:59:38.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.321+0000 7fe77a8d1700 1 --2- 172.21.15.78:0/4091904686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe77406cc00 0x7fe77414c1e0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:38.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.321+0000 7fe77a8d1700 1 --2- 172.21.15.78:0/4091904686 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe75c003a30 0x7fe7740ad7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:38.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.321+0000 7fe77a8d1700 1 --2- 172.21.15.78:0/4091904686 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe76006af00 0x7fe76006d3c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:38.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.321+0000 7fe77a8d1700 1 --2- 172.21.15.78:0/4091904686 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe7741ed490 0x7fe77414bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:38.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.321+0000 7fe77a8d1700 1 -- 172.21.15.78:0/4091904686 >> 172.21.15.78:0/4091904686 conn(0x7fe7740737d0 msgr2=0x7fe77414ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:38.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.321+0000 7fe77a8d1700 1 -- 172.21.15.78:0/4091904686 shutdown_connections 2023-10-14T11:59:38.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:38.321+0000 7fe77a8d1700 1 -- 172.21.15.78:0/4091904686 wait complete. 2023-10-14T11:59:38.364 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 17, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-14T11:59:01.562764+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 17, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "testcephfs1", "in": [ 0 ], "info": { "gid_5031": { "addr": "172.21.15.195:6835/2130671508", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6833", "nonce": 2130671508, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2130671508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5031, "incarnation": 13, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2023-10-14T11:59:04.592935+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5031 } } } ], "standbys": [ { "addr": "172.21.15.195:6834/1492486786", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6832", "nonce": 1492486786, "type": "v2" }, { "addr": "172.21.15.195:6834", "nonce": 1492486786, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4658, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6836/2991384297", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6834", "nonce": 2991384297, "type": "v2" }, { "addr": "172.21.15.78:6836", "nonce": 2991384297, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5016, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6837/3246640037", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6835", "nonce": 3246640037, "type": "v2" }, { "addr": "172.21.15.78:6837", "nonce": 3246640037, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5019, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-14T11:59:38.366 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 2, 'name': 'b', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2023-10-14T11:59:38.366 INFO:tasks.cephfs.test_failover:current = [{'gid': 5019, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6837/3246640037', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 3246640037}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 3246640037}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 4658, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.195:6834/1492486786', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6832', 'nonce': 1492486786}, {'type': 'v1', 'addr': '172.21.15.195:6834', 'nonce': 1492486786}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 5016, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6836/2991384297', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 2991384297}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 2991384297}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 5031, 'name': 'd', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.195:6835/2130671508', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6833', 'nonce': 2130671508}, {'type': 'v1', 'addr': '172.21.15.195:6835', 'nonce': 2130671508}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2023-10-14T11:59:38.367 DEBUG:tasks.cephfs.test_failover:2 != -1 2023-10-14T11:59:38.367 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2023-10-14T11:59:43.369 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T11:59:43.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.504+0000 7f8dd0da7700 1 Processor -- start 2023-10-14T11:59:43.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.504+0000 7f8dd0da7700 1 -- start start 2023-10-14T11:59:43.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.505+0000 7f8dd0da7700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8dcc147f90 0x7f8dcc071370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:43.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.505+0000 7f8dd0da7700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f8dcc0c1430 0x7f8dcc1529f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:43.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.505+0000 7f8dd0da7700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8dcc0c2630 0x7f8dcc154ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:43.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.505+0000 7f8dd0da7700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dcc151500 con 0x7f8dcc147f90 2023-10-14T11:59:43.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.505+0000 7f8dd0da7700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dcc0c1940 con 0x7f8dcc0c1430 2023-10-14T11:59:43.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.505+0000 7f8dd0da7700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dcc151710 con 0x7f8dcc0c2630 2023-10-14T11:59:43.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.505+0000 7f8dca59c700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8dcc147f90 0x7f8dcc071370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:43.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.505+0000 7f8dca59c700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8dcc147f90 0x7f8dcc071370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46768/0 (socket says 172.21.15.78:46768) 2023-10-14T11:59:43.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.505+0000 7f8dca59c700 1 -- 172.21.15.78:0/3702019385 learned_addr learned my addr 172.21.15.78:0/3702019385 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:43.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.506+0000 7f8dc9d9b700 1 --2- 172.21.15.78:0/3702019385 >> v2:172.21.15.195:3300/0 conn(0x7f8dcc0c1430 0x7f8dcc1529f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:43.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.506+0000 7f8dcad9d700 1 --2- 172.21.15.78:0/3702019385 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8dcc0c2630 0x7f8dcc154ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:43.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.506+0000 7f8dca59c700 1 -- 172.21.15.78:0/3702019385 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8dcc0c2630 msgr2=0x7f8dcc154ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:43.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.506+0000 7f8dca59c700 1 --2- 172.21.15.78:0/3702019385 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8dcc0c2630 0x7f8dcc154ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:43.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.506+0000 7f8dca59c700 1 -- 172.21.15.78:0/3702019385 >> v2:172.21.15.195:3300/0 conn(0x7f8dcc0c1430 msgr2=0x7f8dcc1529f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:43.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.506+0000 7f8dca59c700 1 --2- 172.21.15.78:0/3702019385 >> v2:172.21.15.195:3300/0 conn(0x7f8dcc0c1430 0x7f8dcc1529f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:43.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.506+0000 7f8dca59c700 1 -- 172.21.15.78:0/3702019385 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dcc148370 con 0x7f8dcc147f90 2023-10-14T11:59:43.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.506+0000 7f8dca59c700 1 --2- 172.21.15.78:0/3702019385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8dcc147f90 0x7f8dcc071370 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc0a9260 tx=0x7f8db4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b6b322307a5275d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:43.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.506+0000 7f8dc959a700 1 -- 172.21.15.78:0/3702019385 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db4011830 con 0x7f8dcc147f90 2023-10-14T11:59:43.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.506+0000 7f8dc959a700 1 -- 172.21.15.78:0/3702019385 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db4011990 con 0x7f8dcc147f90 2023-10-14T11:59:43.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.506+0000 7f8dc959a700 1 -- 172.21.15.78:0/3702019385 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db4011d20 con 0x7f8dcc147f90 2023-10-14T11:59:43.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.507+0000 7f8dd0da7700 1 -- 172.21.15.78:0/3702019385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8dcc147f90 msgr2=0x7f8dcc071370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:43.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.507+0000 7f8dd0da7700 1 --2- 172.21.15.78:0/3702019385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8dcc147f90 0x7f8dcc071370 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc0a9260 tx=0x7f8db4009bd0 comp rx=0 tx=0).stop 2023-10-14T11:59:43.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.507+0000 7f8dd0da7700 1 -- 172.21.15.78:0/3702019385 shutdown_connections 2023-10-14T11:59:43.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.507+0000 7f8dd0da7700 1 --2- 172.21.15.78:0/3702019385 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8dcc0c2630 0x7f8dcc154ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:43.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.507+0000 7f8dd0da7700 1 --2- 172.21.15.78:0/3702019385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8dcc147f90 0x7f8dcc071370 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:43.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.507+0000 7f8dd0da7700 1 --2- 172.21.15.78:0/3702019385 >> v2:172.21.15.195:3300/0 conn(0x7f8dcc0c1430 0x7f8dcc1529f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:43.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.507+0000 7f8dd0da7700 1 -- 172.21.15.78:0/3702019385 >> 172.21.15.78:0/3702019385 conn(0x7f8dcc0c6fc0 msgr2=0x7f8dcc0c73d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:43.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.507+0000 7f8dd0da7700 1 -- 172.21.15.78:0/3702019385 shutdown_connections 2023-10-14T11:59:43.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.507+0000 7f8dd0da7700 1 -- 172.21.15.78:0/3702019385 wait complete. 2023-10-14T11:59:43.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.508+0000 7f8dd0da7700 1 Processor -- start 2023-10-14T11:59:43.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.508+0000 7f8dd0da7700 1 -- start start 2023-10-14T11:59:43.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.509+0000 7f8dd0da7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8dcc0c1430 0x7f8dcc144c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:43.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.509+0000 7f8dd0da7700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8dcc0c2630 0x7f8dcc06ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:43.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.509+0000 7f8dd0da7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8dcc147f90 0x7f8dcc1ece80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:43.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.509+0000 7f8dd0da7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dcc1e96e0 con 0x7f8dcc0c1430 2023-10-14T11:59:43.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.509+0000 7f8dd0da7700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dcc1e99a0 con 0x7f8dcc147f90 2023-10-14T11:59:43.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.509+0000 7f8dd0da7700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dcc1e9c80 con 0x7f8dcc0c2630 2023-10-14T11:59:43.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.509+0000 7f8dca59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8dcc0c1430 0x7f8dcc144c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:43.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.509+0000 7f8dca59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8dcc0c1430 0x7f8dcc144c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46774/0 (socket says 172.21.15.78:46774) 2023-10-14T11:59:43.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.509+0000 7f8dca59c700 1 -- 172.21.15.78:0/3953740587 learned_addr learned my addr 172.21.15.78:0/3953740587 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:43.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.509+0000 7f8dcad9d700 1 --2- 172.21.15.78:0/3953740587 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8dcc147f90 0x7f8dcc1ece80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:43.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.509+0000 7f8dc9d9b700 1 --2- 172.21.15.78:0/3953740587 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8dcc0c2630 0x7f8dcc06ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:43.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.509+0000 7f8dca59c700 1 -- 172.21.15.78:0/3953740587 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8dcc0c2630 msgr2=0x7f8dcc06ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:43.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.509+0000 7f8dca59c700 1 --2- 172.21.15.78:0/3953740587 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8dcc0c2630 0x7f8dcc06ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:43.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.509+0000 7f8dca59c700 1 -- 172.21.15.78:0/3953740587 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8dcc147f90 msgr2=0x7f8dcc1ece80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:43.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.509+0000 7f8dca59c700 1 --2- 172.21.15.78:0/3953740587 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8dcc147f90 0x7f8dcc1ece80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:43.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.509+0000 7f8dca59c700 1 -- 172.21.15.78:0/3953740587 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db4009550 con 0x7f8dcc0c1430 2023-10-14T11:59:43.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.510+0000 7f8dca59c700 1 --2- 172.21.15.78:0/3953740587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8dcc0c1430 0x7f8dcc144c40 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f8db4009c10 tx=0x7f8db4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:43.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.510+0000 7f8dc2ffd700 1 -- 172.21.15.78:0/3953740587 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db4004450 con 0x7f8dcc0c1430 2023-10-14T11:59:43.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.510+0000 7f8dc2ffd700 1 -- 172.21.15.78:0/3953740587 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db401e070 con 0x7f8dcc0c1430 2023-10-14T11:59:43.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.510+0000 7f8dd0da7700 1 -- 172.21.15.78:0/3953740587 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dcc1e9f60 con 0x7f8dcc0c1430 2023-10-14T11:59:43.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.510+0000 7f8dc2ffd700 1 -- 172.21.15.78:0/3953740587 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db40038e0 con 0x7f8dcc0c1430 2023-10-14T11:59:43.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.510+0000 7f8dd0da7700 1 -- 172.21.15.78:0/3953740587 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dcc1487a0 con 0x7f8dcc0c1430 2023-10-14T11:59:43.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.511+0000 7f8dd0da7700 1 -- 172.21.15.78:0/3953740587 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dcc1eccd0 con 0x7f8dcc0c1430 2023-10-14T11:59:43.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.511+0000 7f8dc2ffd700 1 -- 172.21.15.78:0/3953740587 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8db4003a40 con 0x7f8dcc0c1430 2023-10-14T11:59:43.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.512+0000 7f8dc2ffd700 1 --2- 172.21.15.78:0/3953740587 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8dac06af00 0x7f8dac06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:43.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.512+0000 7f8dc9d9b700 1 --2- 172.21.15.78:0/3953740587 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8dac06af00 0x7f8dac06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:43.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.512+0000 7f8dc2ffd700 1 -- 172.21.15.78:0/3953740587 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f8db408a2b0 con 0x7f8dcc0c1430 2023-10-14T11:59:43.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.512+0000 7f8dc9d9b700 1 --2- 172.21.15.78:0/3953740587 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8dac06af00 0x7f8dac06d3c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc14b100 tx=0x7f8dbc009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:43.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.516+0000 7f8dc2ffd700 1 -- 172.21.15.78:0/3953740587 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8db40553d0 con 0x7f8dcc0c1430 2023-10-14T11:59:43.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.747+0000 7f8dd0da7700 1 -- 172.21.15.78:0/3953740587 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8dcc074b80 con 0x7f8dcc0c1430 2023-10-14T11:59:43.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.747+0000 7f8dc2ffd700 1 -- 172.21.15.78:0/3953740587 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4924 (secure 0 0 0) 0x7f8db40589f0 con 0x7f8dcc0c1430 2023-10-14T11:59:43.750 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:59:43.750 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4658,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6834/1492486786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1492486786},{"type":"v1","addr":"172.21.15.195:6834","nonce":1492486786}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5016,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/2991384297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2991384297},{"type":"v1","addr":"172.21.15.78:6836","nonce":2991384297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5019,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3246640037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3246640037},{"type":"v1","addr":"172.21.15.78:6837","nonce":3246640037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T11:59:01.562764+0000","modified":"2023-10-14T11:59:04.592935+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5031":{"gid":5031,"name":"d","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.195:6835/2130671508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6833","nonce":2130671508},{"type":"v1","addr":"172.21.15.195:6835","nonce":2130671508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"testcephfs1","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-14T11:59:43.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.749+0000 7f8dd0da7700 1 -- 172.21.15.78:0/3953740587 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8dac06af00 msgr2=0x7f8dac06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:43.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.749+0000 7f8dd0da7700 1 --2- 172.21.15.78:0/3953740587 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8dac06af00 0x7f8dac06d3c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc14b100 tx=0x7f8dbc009210 comp rx=0 tx=0).stop 2023-10-14T11:59:43.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.750+0000 7f8dd0da7700 1 -- 172.21.15.78:0/3953740587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8dcc0c1430 msgr2=0x7f8dcc144c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:43.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.750+0000 7f8dd0da7700 1 --2- 172.21.15.78:0/3953740587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8dcc0c1430 0x7f8dcc144c40 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f8db4009c10 tx=0x7f8db4004750 comp rx=0 tx=0).stop 2023-10-14T11:59:43.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.750+0000 7f8dd0da7700 1 -- 172.21.15.78:0/3953740587 shutdown_connections 2023-10-14T11:59:43.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.750+0000 7f8dd0da7700 1 --2- 172.21.15.78:0/3953740587 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8dac06af00 0x7f8dac06d3c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:43.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.750+0000 7f8dd0da7700 1 --2- 172.21.15.78:0/3953740587 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8dcc147f90 0x7f8dcc1ece80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:43.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.750+0000 7f8dd0da7700 1 --2- 172.21.15.78:0/3953740587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8dcc0c1430 0x7f8dcc144c40 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:43.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.750+0000 7f8dd0da7700 1 --2- 172.21.15.78:0/3953740587 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8dcc0c2630 0x7f8dcc06ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:43.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.750+0000 7f8dd0da7700 1 -- 172.21.15.78:0/3953740587 >> 172.21.15.78:0/3953740587 conn(0x7f8dcc0c6fc0 msgr2=0x7f8dcc150da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:43.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.750+0000 7f8dd0da7700 1 -- 172.21.15.78:0/3953740587 shutdown_connections 2023-10-14T11:59:43.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:43.750+0000 7f8dd0da7700 1 -- 172.21.15.78:0/3953740587 wait complete. 2023-10-14T11:59:43.755 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 17 2023-10-14T11:59:44.166 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-14T11:59:01.562764+0000', 'modified': '2023-10-14T11:59:04.592935+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5031}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5031': {'gid': 5031, 'name': 'd', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.195:6835/2130671508', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6833', 'nonce': 2130671508}, {'type': 'v1', 'addr': '172.21.15.195:6835', 'nonce': 2130671508}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'testcephfs1', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-14T11:59:44.166 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-14T11:59:44.167 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-14T11:59:44.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.288+0000 7fee0b191700 1 Processor -- start 2023-10-14T11:59:44.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.288+0000 7fee0b191700 1 -- start start 2023-10-14T11:59:44.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.289+0000 7fee0b191700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fee04154550 0x7fee040c8600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:44.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.289+0000 7fee0b191700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee04155a70 con 0x7fee040c73f0 2023-10-14T11:59:44.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.289+0000 7fee0b191700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0406fcb0 con 0x7fee040c6ee0 2023-10-14T11:59:44.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.289+0000 7fee0b191700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee0406ff90 con 0x7fee04154550 2023-10-14T11:59:44.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.289+0000 7fee08f2d700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fee04154550 0x7fee040c8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:44.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.289+0000 7fee0972e700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fee040c73f0 0x7fee04153550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51662/0 (socket says 172.21.15.78:51662) 2023-10-14T11:59:44.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.289+0000 7fee0972e700 1 -- 172.21.15.78:0/1953606824 learned_addr learned my addr 172.21.15.78:0/1953606824 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:59:44.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.289+0000 7fee08f2d700 1 -- 172.21.15.78:0/1953606824 >> v1:172.21.15.195:6789/0 conn(0x7fee040c6ee0 legacy=0x7fee04152e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:44.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.289+0000 7fee08f2d700 1 -- 172.21.15.78:0/1953606824 >> v1:172.21.15.78:6789/0 conn(0x7fee040c73f0 legacy=0x7fee04153550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:44.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.289+0000 7fee037fe700 1 -- 172.21.15.78:0/1953606824 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 660813710 0 0) 0x7fee04155a70 con 0x7fee040c73f0 2023-10-14T11:59:44.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.289+0000 7fee08f2d700 1 -- 172.21.15.78:0/1953606824 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee04070270 con 0x7fee04154550 2023-10-14T11:59:44.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.289+0000 7fee08f2d700 1 --2- 172.21.15.78:0/1953606824 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee04154550 0x7fee040c8600 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fedf8009840 tx=0x7fedf8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d911cad8da0dbc server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:44.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.289+0000 7fee037fe700 1 -- 172.21.15.78:0/1953606824 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf8004230 con 0x7fee04154550 2023-10-14T11:59:44.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.289+0000 7fee037fe700 1 -- 172.21.15.78:0/1953606824 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf8004390 con 0x7fee04154550 2023-10-14T11:59:44.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.289+0000 7fee037fe700 1 -- 172.21.15.78:0/1953606824 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf80036a0 con 0x7fee04154550 2023-10-14T11:59:44.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.290+0000 7fee0b191700 1 -- 172.21.15.78:0/1953606824 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee04154550 msgr2=0x7fee040c8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:44.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.290+0000 7fee0b191700 1 --2- 172.21.15.78:0/1953606824 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee04154550 0x7fee040c8600 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fedf8009840 tx=0x7fedf8009b50 comp rx=0 tx=0).stop 2023-10-14T11:59:44.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.290+0000 7fee0b191700 1 -- 172.21.15.78:0/1953606824 shutdown_connections 2023-10-14T11:59:44.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.290+0000 7fee0b191700 1 --2- 172.21.15.78:0/1953606824 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee04154550 0x7fee040c8600 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:44.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.290+0000 7fee0b191700 1 -- 172.21.15.78:0/1953606824 >> 172.21.15.78:0/1953606824 conn(0x7fee040737d0 msgr2=0x7fee04073be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:44.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.290+0000 7fee0b191700 1 -- 172.21.15.78:0/1953606824 shutdown_connections 2023-10-14T11:59:44.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.290+0000 7fee0b191700 1 -- 172.21.15.78:0/1953606824 wait complete. 2023-10-14T11:59:44.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.291+0000 7fee0b191700 1 Processor -- start 2023-10-14T11:59:44.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.291+0000 7fee0b191700 1 -- start start 2023-10-14T11:59:44.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.292+0000 7fee0b191700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fee040c6ee0 0x7fee04158ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:44.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.292+0000 7fee0b191700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee040c73f0 0x7fee040bc510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:44.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.292+0000 7fee0b191700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee04154550 0x7fee040bca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:44.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.292+0000 7fee0b191700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee040c50c0 con 0x7fee04154550 2023-10-14T11:59:44.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.292+0000 7fee0b191700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee040c54e0 con 0x7fee040c73f0 2023-10-14T11:59:44.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.292+0000 7fee0b191700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee04158ff0 con 0x7fee040c6ee0 2023-10-14T11:59:44.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.292+0000 7fee0972e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee04154550 0x7fee040bca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:44.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.292+0000 7fee0972e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee04154550 0x7fee040bca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46792/0 (socket says 172.21.15.78:46792) 2023-10-14T11:59:44.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.292+0000 7fee0972e700 1 -- 172.21.15.78:0/2388991947 learned_addr learned my addr 172.21.15.78:0/2388991947 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:44.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.292+0000 7fee03fff700 1 --2- 172.21.15.78:0/2388991947 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee040c73f0 0x7fee040bc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:44.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.292+0000 7fee08f2d700 1 --2- 172.21.15.78:0/2388991947 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fee040c6ee0 0x7fee04158ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:44.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.292+0000 7fee0972e700 1 -- 172.21.15.78:0/2388991947 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fee040c6ee0 msgr2=0x7fee04158ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:44.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.292+0000 7fee0972e700 1 --2- 172.21.15.78:0/2388991947 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fee040c6ee0 0x7fee04158ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:44.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.292+0000 7fee0972e700 1 -- 172.21.15.78:0/2388991947 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee040c73f0 msgr2=0x7fee040bc510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:44.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.293+0000 7fee0972e700 1 --2- 172.21.15.78:0/2388991947 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee040c73f0 0x7fee040bc510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:44.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.293+0000 7fee0972e700 1 -- 172.21.15.78:0/2388991947 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf80094a0 con 0x7fee04154550 2023-10-14T11:59:44.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.293+0000 7fee0972e700 1 --2- 172.21.15.78:0/2388991947 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee04154550 0x7fee040bca50 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00cca0 tx=0x7fedfc00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:44.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.293+0000 7fee017fa700 1 -- 172.21.15.78:0/2388991947 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc00cea0 con 0x7fee04154550 2023-10-14T11:59:44.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.293+0000 7fee017fa700 1 -- 172.21.15.78:0/2388991947 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedfc00ade0 con 0x7fee04154550 2023-10-14T11:59:44.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.293+0000 7fee0b191700 1 -- 172.21.15.78:0/2388991947 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee040c4440 con 0x7fee04154550 2023-10-14T11:59:44.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.293+0000 7fee017fa700 1 -- 172.21.15.78:0/2388991947 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc007310 con 0x7fee04154550 2023-10-14T11:59:44.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.293+0000 7fee0b191700 1 -- 172.21.15.78:0/2388991947 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee040c4a00 con 0x7fee04154550 2023-10-14T11:59:44.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.294+0000 7fee017fa700 1 -- 172.21.15.78:0/2388991947 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fedfc007470 con 0x7fee04154550 2023-10-14T11:59:44.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.294+0000 7fee017fa700 1 --2- 172.21.15.78:0/2388991947 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fede806af00 0x7fede806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:44.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.295+0000 7fee017fa700 1 -- 172.21.15.78:0/2388991947 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7fedfc089520 con 0x7fee04154550 2023-10-14T11:59:44.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.295+0000 7fee08f2d700 1 --2- 172.21.15.78:0/2388991947 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fede806af00 0x7fede806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:44.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.295+0000 7fee08f2d700 1 --2- 172.21.15.78:0/2388991947 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fede806af00 0x7fede806d3c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fedf8009b90 tx=0x7fedf8013040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:44.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.296+0000 7fee1737bb80 1 Processor -- start 2023-10-14T11:59:44.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.297+0000 7fee1737bb80 1 -- start start 2023-10-14T11:59:44.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.297+0000 7fee1737bb80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x555a6cdc36b0 0x555a6ce01620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:44.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.297+0000 7fee1737bb80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x555a6cdc3a90 0x555a6ce01b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:44.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.297+0000 7fee1737bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a6cdbdc90 0x555a6cde18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:44.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.297+0000 7fee1737bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a6cd37c20 con 0x555a6cdbdc90 2023-10-14T11:59:44.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.297+0000 7fee1737bb80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a6cd1cc10 con 0x555a6cdc3a90 2023-10-14T11:59:44.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.297+0000 7fee1737bb80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x555a6cd1a530 con 0x555a6cdc36b0 2023-10-14T11:59:44.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.297+0000 7fee0972e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a6cdbdc90 0x555a6cde18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:44.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.297+0000 7fee0972e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a6cdbdc90 0x555a6cde18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46800/0 (socket says 172.21.15.78:46800) 2023-10-14T11:59:44.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.297+0000 7fee0972e700 1 -- 172.21.15.78:0/3857090173 learned_addr learned my addr 172.21.15.78:0/3857090173 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:44.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.297+0000 7fee03fff700 1 --2- 172.21.15.78:0/3857090173 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x555a6cdc3a90 0x555a6ce01b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:44.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.297+0000 7fee08f2d700 1 --2- 172.21.15.78:0/3857090173 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x555a6cdc36b0 0x555a6ce01620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:44.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.298+0000 7fee0972e700 1 -- 172.21.15.78:0/3857090173 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x555a6cdc36b0 msgr2=0x555a6ce01620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:44.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.298+0000 7fee0972e700 1 --2- 172.21.15.78:0/3857090173 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x555a6cdc36b0 0x555a6ce01620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:44.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.298+0000 7fee0972e700 1 -- 172.21.15.78:0/3857090173 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x555a6cdc3a90 msgr2=0x555a6ce01b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:44.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.298+0000 7fee0972e700 1 --2- 172.21.15.78:0/3857090173 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x555a6cdc3a90 0x555a6ce01b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:44.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.298+0000 7fee0972e700 1 -- 172.21.15.78:0/3857090173 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a6ccd33a0 con 0x555a6cdbdc90 2023-10-14T11:59:44.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.298+0000 7fee0972e700 1 --2- 172.21.15.78:0/3857090173 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a6cdbdc90 0x555a6cde18a0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00cc70 tx=0x7fedfc0546f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c3a3ec796f7f36d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:44.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.298+0000 7fedee7fc700 1 -- 172.21.15.78:0/3857090173 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc05c450 con 0x555a6cdbdc90 2023-10-14T11:59:44.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.298+0000 7fedee7fc700 1 -- 172.21.15.78:0/3857090173 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedfc060600 con 0x555a6cdbdc90 2023-10-14T11:59:44.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.298+0000 7fedee7fc700 1 -- 172.21.15.78:0/3857090173 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc05c160 con 0x555a6cdbdc90 2023-10-14T11:59:44.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.298+0000 7fee1737bb80 1 -- 172.21.15.78:0/3857090173 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a6cdbdc90 msgr2=0x555a6cde18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:44.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.298+0000 7fee1737bb80 1 --2- 172.21.15.78:0/3857090173 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a6cdbdc90 0x555a6cde18a0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00cc70 tx=0x7fedfc0546f0 comp rx=0 tx=0).stop 2023-10-14T11:59:44.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.298+0000 7fee1737bb80 1 -- 172.21.15.78:0/3857090173 shutdown_connections 2023-10-14T11:59:44.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.298+0000 7fee1737bb80 1 --2- 172.21.15.78:0/3857090173 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a6cdbdc90 0x555a6cde18a0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:44.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.298+0000 7fee1737bb80 1 --2- 172.21.15.78:0/3857090173 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x555a6cdc36b0 0x555a6ce01620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:44.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.298+0000 7fee1737bb80 1 --2- 172.21.15.78:0/3857090173 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x555a6cdc3a90 0x555a6ce01b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:44.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.298+0000 7fee1737bb80 1 -- 172.21.15.78:0/3857090173 >> 172.21.15.78:0/3857090173 conn(0x555a6ccf0d80 msgr2=0x555a6cdc13a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:44.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.299+0000 7fee1737bb80 1 -- 172.21.15.78:0/3857090173 shutdown_connections 2023-10-14T11:59:44.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.299+0000 7fee1737bb80 1 -- 172.21.15.78:0/3857090173 wait complete. 2023-10-14T11:59:44.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.299+0000 7fee1737bb80 1 -- start start 2023-10-14T11:59:44.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.299+0000 7fee1737bb80 1 Processor -- start 2023-10-14T11:59:44.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.300+0000 7fee1737bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a6cdbdc90 0x555a6cdfe9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:44.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.300+0000 7fee1737bb80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x555a6cdc36b0 0x555a6cdfd590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:44.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.300+0000 7fee1737bb80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x555a6cdc3a90 0x555a6ce12330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:44.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.300+0000 7fee1737bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a6cd15e20 con 0x555a6cdbdc90 2023-10-14T11:59:44.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.300+0000 7fee1737bb80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a6ccb26c0 con 0x555a6cdc36b0 2023-10-14T11:59:44.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.300+0000 7fee1737bb80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x555a6ccb2360 con 0x555a6cdc3a90 2023-10-14T11:59:44.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.301+0000 7fee1737bb80 10 client.0 _ll_register_callbacks cb 0x555a6cb0f7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T11:59:44.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.301+0000 7fee08f2d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a6cdbdc90 0x555a6cdfe9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:44.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.301+0000 7fee08f2d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a6cdbdc90 0x555a6cdfe9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46802/0 (socket says 172.21.15.78:46802) 2023-10-14T11:59:44.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.301+0000 7fee08f2d700 1 -- 172.21.15.78:0/4042724222 learned_addr learned my addr 172.21.15.78:0/4042724222 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:44.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.301+0000 7fee03fff700 1 --2- 172.21.15.78:0/4042724222 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x555a6cdc36b0 0x555a6cdfd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:44.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.301+0000 7fee0972e700 1 --2- 172.21.15.78:0/4042724222 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x555a6cdc3a90 0x555a6ce12330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:44.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.301+0000 7fee08f2d700 1 -- 172.21.15.78:0/4042724222 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x555a6cdc3a90 msgr2=0x555a6ce12330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:44.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.301+0000 7fee08f2d700 1 --2- 172.21.15.78:0/4042724222 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x555a6cdc3a90 0x555a6ce12330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:44.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.301+0000 7fee08f2d700 1 -- 172.21.15.78:0/4042724222 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x555a6cdc36b0 msgr2=0x555a6cdfd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:44.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.301+0000 7fee08f2d700 1 --2- 172.21.15.78:0/4042724222 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x555a6cdc36b0 0x555a6cdfd590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:44.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.301+0000 7fee08f2d700 1 -- 172.21.15.78:0/4042724222 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee040c4a00 con 0x555a6cdbdc90 2023-10-14T11:59:44.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.301+0000 7fee08f2d700 1 --2- 172.21.15.78:0/4042724222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a6cdbdc90 0x555a6cdfe9b0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fedf8004000 tx=0x7fedf80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ceaad505288a6dc server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:44.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.301+0000 7fede37fe700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-14T11:59:44.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.301+0000 7fede37fe700 1 -- 172.21.15.78:0/4042724222 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf80118c0 con 0x555a6cdbdc90 2023-10-14T11:59:44.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.301+0000 7fede37fe700 1 -- 172.21.15.78:0/4042724222 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7fedf8011a20 con 0x555a6cdbdc90 2023-10-14T11:59:44.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.301+0000 7fee1737bb80 1 -- 172.21.15.78:0/4042724222 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555a6cdb65b0 con 0x555a6cdbdc90 2023-10-14T11:59:44.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.301+0000 7fede37fe700 1 -- 172.21.15.78:0/4042724222 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fedf8011be0 con 0x555a6cdbdc90 2023-10-14T11:59:44.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.301+0000 7fee1737bb80 10 client.5397 fetch_fsmap learned FSMap version 17 2023-10-14T11:59:44.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.301+0000 7fee1737bb80 1 -- 172.21.15.78:0/4042724222 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x555a6cb40d90 con 0x555a6cdbdc90 2023-10-14T11:59:44.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.301+0000 7fede37fe700 1 -- 172.21.15.78:0/4042724222 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x555a6cdb65b0 con 0x555a6cdbdc90 2023-10-14T11:59:44.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.301+0000 7fede37fe700 1 -- 172.21.15.78:0/4042724222 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 17) v1 ==== 3299+0+0 (secure 0 0 0) 0x7fedf802a480 con 0x555a6cdbdc90 2023-10-14T11:59:44.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.301+0000 7fee1737bb80 10 client.5397 fetch_fsmap finished waiting for FSMap version 17 2023-10-14T11:59:44.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.302+0000 7fee1737bb80 10 client.5397 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-14T11:59:44.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.302+0000 7fee1737bb80 20 client.5397 populate_metadata read hostname 'smithi078' 2023-10-14T11:59:44.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.302+0000 7fee1737bb80 1 --2- 172.21.15.78:0/4042724222 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x555a6cde6590 0x555a6ce12870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:44.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.302+0000 7fee1737bb80 4 client.5397 mds_command: new command op to 5031 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-14T11:59:44.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.302+0000 7fee1737bb80 1 -- 172.21.15.78:0/4042724222 --> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555a6cd0ee00 con 0x555a6cde6590 2023-10-14T11:59:44.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.302+0000 7fee03fff700 1 --2- 172.21.15.78:0/4042724222 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x555a6cde6590 0x555a6ce12870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:44.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.302+0000 7fede37fe700 10 client.5397 ms_handle_connect on v2:172.21.15.195:6833/2130671508 2023-10-14T11:59:44.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.302+0000 7fee03fff700 1 --2- 172.21.15.78:0/4042724222 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x555a6cde6590 0x555a6ce12870 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=680497ce6b044022 server_cookie=ede104f6e89b662 in_seq=0 out_seq=0 2023-10-14T11:59:44.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.303+0000 7fede37fe700 1 -- 172.21.15.78:0/4042724222 <== mds.0 v2:172.21.15.195:6833/2130671508 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x555a6cd0ee00 con 0x555a6cde6590 2023-10-14T11:59:44.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.303+0000 7fede37fe700 10 client.5397 handle_command_reply: tid=0 2023-10-14T11:59:44.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.303+0000 7fee1737bb80 1 client.5397 shutdown 2023-10-14T11:59:44.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.303+0000 7fee1737bb80 1 -- 172.21.15.78:0/4042724222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a6cdbdc90 msgr2=0x555a6cdfe9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:44.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.303+0000 7fee1737bb80 1 --2- 172.21.15.78:0/4042724222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a6cdbdc90 0x555a6cdfe9b0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fedf8004000 tx=0x7fedf80047c0 comp rx=0 tx=0).stop 2023-10-14T11:59:44.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.303+0000 7fee1737bb80 1 -- 172.21.15.78:0/4042724222 shutdown_connections 2023-10-14T11:59:44.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.303+0000 7fee1737bb80 1 --2- 172.21.15.78:0/4042724222 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x555a6cdc3a90 0x555a6ce12330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:44.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.303+0000 7fee1737bb80 1 --2- 172.21.15.78:0/4042724222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a6cdbdc90 0x555a6cdfe9b0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:44.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.303+0000 7fee1737bb80 1 --2- 172.21.15.78:0/4042724222 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x555a6cde6590 0x555a6ce12870 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:44.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.303+0000 7fee1737bb80 1 --2- 172.21.15.78:0/4042724222 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x555a6cdc36b0 0x555a6cdfd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:44.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.303+0000 7fede37fe700 0 client.5397 ms_handle_reset on v2:172.21.15.195:6833/2130671508 2023-10-14T11:59:44.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.303+0000 7fee1737bb80 1 -- 172.21.15.78:0/4042724222 >> 172.21.15.78:0/4042724222 conn(0x555a6ccf0d80 msgr2=0x555a6cdb87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:44.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.303+0000 7fee1737bb80 1 -- 172.21.15.78:0/4042724222 shutdown_connections 2023-10-14T11:59:44.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.303+0000 7fee1737bb80 1 -- 172.21.15.78:0/4042724222 wait complete. 2023-10-14T11:59:44.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.303+0000 7fee1737bb80 20 client.5397 trim_cache size 0 max 16384 2023-10-14T11:59:44.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.321+0000 7fee1737bb80 1 Processor -- start 2023-10-14T11:59:44.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.321+0000 7fee1737bb80 1 -- start start 2023-10-14T11:59:44.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.321+0000 7fee1737bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a6cdbdc90 0x555a6cde4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:44.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.321+0000 7fee1737bb80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x555a6cdc36b0 0x555a6cde5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:44.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.321+0000 7fee1737bb80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x555a6cdc3a90 0x555a6cdfd510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:44.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.321+0000 7fee1737bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a6cb968d0 con 0x555a6cdbdc90 2023-10-14T11:59:44.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.321+0000 7fee1737bb80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a6cdff150 con 0x555a6cdc36b0 2023-10-14T11:59:44.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.321+0000 7fee1737bb80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x555a6ce0c960 con 0x555a6cdc3a90 2023-10-14T11:59:44.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.321+0000 7fee08f2d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a6cdbdc90 0x555a6cde4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:44.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.322+0000 7fee08f2d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a6cdbdc90 0x555a6cde4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46810/0 (socket says 172.21.15.78:46810) 2023-10-14T11:59:44.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.322+0000 7fee08f2d700 1 -- 172.21.15.78:0/4291351999 learned_addr learned my addr 172.21.15.78:0/4291351999 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:44.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.322+0000 7fee03fff700 1 --2- 172.21.15.78:0/4291351999 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x555a6cdc36b0 0x555a6cde5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:44.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.322+0000 7fee0972e700 1 --2- 172.21.15.78:0/4291351999 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x555a6cdc3a90 0x555a6cdfd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:44.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.322+0000 7fee08f2d700 1 -- 172.21.15.78:0/4291351999 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x555a6cdc3a90 msgr2=0x555a6cdfd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:44.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.322+0000 7fee08f2d700 1 --2- 172.21.15.78:0/4291351999 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x555a6cdc3a90 0x555a6cdfd510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:44.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.322+0000 7fee08f2d700 1 -- 172.21.15.78:0/4291351999 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x555a6cdc36b0 msgr2=0x555a6cde5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:44.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.322+0000 7fee08f2d700 1 --2- 172.21.15.78:0/4291351999 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x555a6cdc36b0 0x555a6cde5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:44.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.322+0000 7fee08f2d700 1 -- 172.21.15.78:0/4291351999 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee040bd0b0 con 0x555a6cdbdc90 2023-10-14T11:59:44.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.322+0000 7fee03fff700 1 --2- 172.21.15.78:0/4291351999 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x555a6cdc36b0 0x555a6cde5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:59:44.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.322+0000 7fee08f2d700 1 --2- 172.21.15.78:0/4291351999 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a6cdbdc90 0x555a6cde4ed0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fedf8009840 tx=0x7fedf8022070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dc3d3642fc89d45 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:44.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.322+0000 7fedeeffd700 1 -- 172.21.15.78:0/4291351999 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf802a480 con 0x555a6cdbdc90 2023-10-14T11:59:44.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.322+0000 7fedeeffd700 1 -- 172.21.15.78:0/4291351999 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf802a5e0 con 0x555a6cdbdc90 2023-10-14T11:59:44.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.322+0000 7fedeeffd700 1 -- 172.21.15.78:0/4291351999 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf802ac20 con 0x555a6cdbdc90 2023-10-14T11:59:44.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.323+0000 7fee1737bb80 1 -- 172.21.15.78:0/4291351999 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a6cdbdc90 msgr2=0x555a6cde4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:44.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.323+0000 7fee1737bb80 1 --2- 172.21.15.78:0/4291351999 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a6cdbdc90 0x555a6cde4ed0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fedf8009840 tx=0x7fedf8022070 comp rx=0 tx=0).stop 2023-10-14T11:59:44.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.323+0000 7fee1737bb80 1 -- 172.21.15.78:0/4291351999 shutdown_connections 2023-10-14T11:59:44.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.323+0000 7fee1737bb80 1 --2- 172.21.15.78:0/4291351999 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x555a6cdc3a90 0x555a6cdfd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:44.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.323+0000 7fee1737bb80 1 --2- 172.21.15.78:0/4291351999 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a6cdbdc90 0x555a6cde4ed0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:44.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.323+0000 7fee1737bb80 1 --2- 172.21.15.78:0/4291351999 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x555a6cdc36b0 0x555a6cde5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:44.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.323+0000 7fee1737bb80 1 -- 172.21.15.78:0/4291351999 >> 172.21.15.78:0/4291351999 conn(0x555a6ccf0d80 msgr2=0x555a6cdc1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:44.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.323+0000 7fee1737bb80 1 -- 172.21.15.78:0/4291351999 shutdown_connections 2023-10-14T11:59:44.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.323+0000 7fee1737bb80 1 -- 172.21.15.78:0/4291351999 wait complete. 2023-10-14T11:59:44.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.323+0000 7fee1737bb80 1 -- start start 2023-10-14T11:59:44.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.323+0000 7fee1737bb80 1 Processor -- start 2023-10-14T11:59:44.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.324+0000 7fee1737bb80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x555a6cdbdc90 0x555a6cde0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:44.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.324+0000 7fee1737bb80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x555a6cdc36b0 0x555a6ce6a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:44.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.324+0000 7fee1737bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a6cdc3a90 0x555a6ce63e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:44.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.324+0000 7fee1737bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a6ce03250 con 0x555a6cdc3a90 2023-10-14T11:59:44.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.324+0000 7fee1737bb80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a6ce0f020 con 0x555a6cdc36b0 2023-10-14T11:59:44.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.324+0000 7fee1737bb80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x555a6ce6a780 con 0x555a6cdbdc90 2023-10-14T11:59:44.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.324+0000 7fee1737bb80 10 client.0 _ll_register_callbacks cb 0x7fedf8004220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T11:59:44.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.324+0000 7fee0972e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a6cdc3a90 0x555a6ce63e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:44.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.324+0000 7fee0972e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a6cdc3a90 0x555a6ce63e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46820/0 (socket says 172.21.15.78:46820) 2023-10-14T11:59:44.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.324+0000 7fee0972e700 1 -- 172.21.15.78:0/1695290261 learned_addr learned my addr 172.21.15.78:0/1695290261 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:44.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.324+0000 7fee03fff700 1 --2- 172.21.15.78:0/1695290261 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x555a6cdc36b0 0x555a6ce6a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:44.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.324+0000 7fee08f2d700 1 --2- 172.21.15.78:0/1695290261 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x555a6cdbdc90 0x555a6cde0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:44.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.324+0000 7fee0972e700 1 -- 172.21.15.78:0/1695290261 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x555a6cdbdc90 msgr2=0x555a6cde0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:44.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.324+0000 7fee0972e700 1 --2- 172.21.15.78:0/1695290261 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x555a6cdbdc90 0x555a6cde0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:44.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.324+0000 7fee0972e700 1 -- 172.21.15.78:0/1695290261 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x555a6cdc36b0 msgr2=0x555a6ce6a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:44.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.324+0000 7fee0972e700 1 --2- 172.21.15.78:0/1695290261 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x555a6cdc36b0 0x555a6ce6a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:44.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.324+0000 7fee0972e700 1 -- 172.21.15.78:0/1695290261 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555a6cb40d90 con 0x555a6cdc3a90 2023-10-14T11:59:44.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.325+0000 7fee0972e700 1 --2- 172.21.15.78:0/1695290261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a6cdc3a90 0x555a6ce63e60 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00cc70 tx=0x7fedfc089070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be2a313d8f67c62d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:44.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.325+0000 7fede37fe700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-14T11:59:44.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.325+0000 7fede37fe700 1 -- 172.21.15.78:0/1695290261 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc001c90 con 0x555a6cdc3a90 2023-10-14T11:59:44.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.325+0000 7fede37fe700 1 -- 172.21.15.78:0/1695290261 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7fedfc06f040 con 0x555a6cdc3a90 2023-10-14T11:59:44.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.325+0000 7fee1737bb80 1 -- 172.21.15.78:0/1695290261 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fedfc060600 con 0x555a6cdc3a90 2023-10-14T11:59:44.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.325+0000 7fede37fe700 1 -- 172.21.15.78:0/1695290261 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fedfc0580d0 con 0x555a6cdc3a90 2023-10-14T11:59:44.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.325+0000 7fee1737bb80 10 client.5403 fetch_fsmap learned FSMap version 17 2023-10-14T11:59:44.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.325+0000 7fee1737bb80 1 -- 172.21.15.78:0/1695290261 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x555a6cb29c10 con 0x555a6cdc3a90 2023-10-14T11:59:44.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.325+0000 7fede37fe700 1 -- 172.21.15.78:0/1695290261 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedfc060600 con 0x555a6cdc3a90 2023-10-14T11:59:44.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.325+0000 7fede37fe700 1 -- 172.21.15.78:0/1695290261 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 17) v1 ==== 3299+0+0 (secure 0 0 0) 0x7fedfc060170 con 0x555a6cdc3a90 2023-10-14T11:59:44.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.325+0000 7fee1737bb80 10 client.5403 fetch_fsmap finished waiting for FSMap version 17 2023-10-14T11:59:44.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.325+0000 7fee1737bb80 10 client.5403 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-14T11:59:44.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.325+0000 7fee1737bb80 20 client.5403 populate_metadata read hostname 'smithi078' 2023-10-14T11:59:44.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.325+0000 7fee1737bb80 1 --2- 172.21.15.78:0/1695290261 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x555a6ce643a0 0x555a6cdf4f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:44.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.325+0000 7fee1737bb80 4 client.5403 mds_command: new command op to 5031 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-14T11:59:44.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.325+0000 7fee1737bb80 1 -- 172.21.15.78:0/1695290261 --> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] -- command(tid 0: {"prefix": "status"}) v1 -- 0x555a6cd0eb20 con 0x555a6ce643a0 2023-10-14T11:59:44.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.325+0000 7fee08f2d700 1 --2- 172.21.15.78:0/1695290261 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x555a6ce643a0 0x555a6cdf4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:44.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.326+0000 7fede37fe700 10 client.5403 ms_handle_connect on v2:172.21.15.195:6833/2130671508 2023-10-14T11:59:44.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.326+0000 7fee08f2d700 1 --2- 172.21.15.78:0/1695290261 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x555a6ce643a0 0x555a6cdf4f10 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1dc30da66a5e85cf server_cookie=31a9379bf25f9d55 in_seq=0 out_seq=0 2023-10-14T11:59:44.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.326+0000 7fede37fe700 1 -- 172.21.15.78:0/1695290261 <== mds.0 v2:172.21.15.195:6833/2130671508 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+336 (crc 0 0 0) 0x555a6cd0eb20 con 0x555a6ce643a0 2023-10-14T11:59:44.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.326+0000 7fede37fe700 10 client.5403 handle_command_reply: tid=0 2023-10-14T11:59:44.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.326+0000 7fee1737bb80 1 client.5403 shutdown 2023-10-14T11:59:44.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.326+0000 7fee1737bb80 1 -- 172.21.15.78:0/1695290261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a6cdc3a90 msgr2=0x555a6ce63e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:44.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.326+0000 7fee1737bb80 1 --2- 172.21.15.78:0/1695290261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a6cdc3a90 0x555a6ce63e60 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00cc70 tx=0x7fedfc089070 comp rx=0 tx=0).stop 2023-10-14T11:59:44.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.326+0000 7fee1737bb80 1 -- 172.21.15.78:0/1695290261 shutdown_connections 2023-10-14T11:59:44.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.326+0000 7fee1737bb80 1 --2- 172.21.15.78:0/1695290261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a6cdc3a90 0x555a6ce63e60 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:44.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.326+0000 7fee1737bb80 1 --2- 172.21.15.78:0/1695290261 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x555a6cdbdc90 0x555a6cde0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:44.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.326+0000 7fee1737bb80 1 --2- 172.21.15.78:0/1695290261 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x555a6ce643a0 0x555a6cdf4f10 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:44.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.326+0000 7fee1737bb80 1 --2- 172.21.15.78:0/1695290261 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x555a6cdc36b0 0x555a6ce6a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:44.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.326+0000 7fede37fe700 0 client.5403 ms_handle_reset on v2:172.21.15.195:6833/2130671508 2023-10-14T11:59:44.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.326+0000 7fee1737bb80 1 -- 172.21.15.78:0/1695290261 >> 172.21.15.78:0/1695290261 conn(0x555a6ccf0d80 msgr2=0x555a6cdfd510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:44.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.326+0000 7fee1737bb80 1 -- 172.21.15.78:0/1695290261 shutdown_connections 2023-10-14T11:59:44.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.326+0000 7fee1737bb80 1 -- 172.21.15.78:0/1695290261 wait complete. 2023-10-14T11:59:44.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.326+0000 7fee1737bb80 20 client.5403 trim_cache size 0 max 16384 2023-10-14T11:59:44.380 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-14T11:59:44.380 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "4c5dc134-3092-406a-a1e1-13e6d9e926e8", 2023-10-14T11:59:44.380 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2023-10-14T11:59:44.380 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 5031, 2023-10-14T11:59:44.381 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2023-10-14T11:59:44.381 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2023-10-14T11:59:44.381 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "testcephfs1", 2023-10-14T11:59:44.381 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 42.142023962000003, 2023-10-14T11:59:44.381 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 17, 2023-10-14T11:59:44.382 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 32, 2023-10-14T11:59:44.382 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 32, 2023-10-14T11:59:44.382 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 50.032846718999998 2023-10-14T11:59:44.382 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-14T11:59:44.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.327+0000 7fee0b191700 1 -- 172.21.15.78:0/2388991947 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fede806af00 msgr2=0x7fede806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:44.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.327+0000 7fee0b191700 1 --2- 172.21.15.78:0/2388991947 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fede806af00 0x7fede806d3c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fedf8009b90 tx=0x7fedf8013040 comp rx=0 tx=0).stop 2023-10-14T11:59:44.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.327+0000 7fee0b191700 1 -- 172.21.15.78:0/2388991947 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee04154550 msgr2=0x7fee040bca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:44.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.327+0000 7fee0b191700 1 --2- 172.21.15.78:0/2388991947 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee04154550 0x7fee040bca50 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00cca0 tx=0x7fedfc00a340 comp rx=0 tx=0).stop 2023-10-14T11:59:44.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.327+0000 7fee0b191700 1 -- 172.21.15.78:0/2388991947 shutdown_connections 2023-10-14T11:59:44.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.327+0000 7fee0b191700 1 --2- 172.21.15.78:0/2388991947 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee04154550 0x7fee040bca50 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:44.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.327+0000 7fee0b191700 1 --2- 172.21.15.78:0/2388991947 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fee040c6ee0 0x7fee04158ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:44.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.327+0000 7fee0b191700 1 --2- 172.21.15.78:0/2388991947 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fede806af00 0x7fede806d3c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:44.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.327+0000 7fee0b191700 1 --2- 172.21.15.78:0/2388991947 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee040c73f0 0x7fee040bc510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:44.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.327+0000 7fee0b191700 1 -- 172.21.15.78:0/2388991947 >> 172.21.15.78:0/2388991947 conn(0x7fee040737d0 msgr2=0x7fee041520a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:44.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.327+0000 7fee0b191700 1 -- 172.21.15.78:0/2388991947 shutdown_connections 2023-10-14T11:59:44.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:44.327+0000 7fee0b191700 1 -- 172.21.15.78:0/2388991947 wait complete. 2023-10-14T11:59:44.386 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 17, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-14T11:59:01.562764+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 17, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "testcephfs1", "in": [ 0 ], "info": { "gid_5031": { "addr": "172.21.15.195:6835/2130671508", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6833", "nonce": 2130671508, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2130671508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5031, "incarnation": 13, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2023-10-14T11:59:04.592935+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5031 } } } ], "standbys": [ { "addr": "172.21.15.195:6834/1492486786", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6832", "nonce": 1492486786, "type": "v2" }, { "addr": "172.21.15.195:6834", "nonce": 1492486786, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4658, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6836/2991384297", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6834", "nonce": 2991384297, "type": "v2" }, { "addr": "172.21.15.78:6836", "nonce": 2991384297, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5016, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6837/3246640037", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6835", "nonce": 3246640037, "type": "v2" }, { "addr": "172.21.15.78:6837", "nonce": 3246640037, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5019, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-14T11:59:44.387 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 2, 'name': 'b', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2023-10-14T11:59:44.388 INFO:tasks.cephfs.test_failover:current = [{'gid': 5019, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6837/3246640037', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 3246640037}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 3246640037}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 4658, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.195:6834/1492486786', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6832', 'nonce': 1492486786}, {'type': 'v1', 'addr': '172.21.15.195:6834', 'nonce': 1492486786}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 5016, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6836/2991384297', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 2991384297}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 2991384297}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 5031, 'name': 'd', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.195:6835/2130671508', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6833', 'nonce': 2130671508}, {'type': 'v1', 'addr': '172.21.15.195:6835', 'nonce': 2130671508}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2023-10-14T11:59:44.388 DEBUG:tasks.cephfs.test_failover:2 != -1 2023-10-14T11:59:44.389 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2023-10-14T11:59:49.389 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T11:59:49.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.524+0000 7f5a9692d700 1 Processor -- start 2023-10-14T11:59:49.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.524+0000 7f5a9692d700 1 -- start start 2023-10-14T11:59:49.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.525+0000 7f5a9692d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f5a900abc90 0x7f5a900c1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:49.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.525+0000 7f5a9692d700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5a9006e360 0x7f5a90156650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:49.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.525+0000 7f5a9692d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a9006db20 con 0x7f5a90156b90 2023-10-14T11:59:49.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.525+0000 7f5a9692d700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a9006ae70 con 0x7f5a9006e360 2023-10-14T11:59:49.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.525+0000 7f5a9692d700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a9006ddc0 con 0x7f5a900abc90 2023-10-14T11:59:49.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.525+0000 7f5a8f7fe700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5a9006e360 0x7f5a90156650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:49.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.525+0000 7f5a8f7fe700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5a9006e360 0x7f5a90156650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46824/0 (socket says 172.21.15.78:46824) 2023-10-14T11:59:49.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.525+0000 7f5a8f7fe700 1 -- 172.21.15.78:0/1530164923 learned_addr learned my addr 172.21.15.78:0/1530164923 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:49.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.525+0000 7f5a8f7fe700 1 -- 172.21.15.78:0/1530164923 >> v2:172.21.15.195:3300/0 conn(0x7f5a900abc90 msgr2=0x7f5a900c1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:49.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.525+0000 7f5a8f7fe700 1 --2- 172.21.15.78:0/1530164923 >> v2:172.21.15.195:3300/0 conn(0x7f5a900abc90 0x7f5a900c1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:49.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.525+0000 7f5a8f7fe700 1 -- 172.21.15.78:0/1530164923 >> v1:172.21.15.195:6789/0 conn(0x7f5a90156b90 legacy=0x7f5a900c2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:49.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.525+0000 7f5a8f7fe700 1 -- 172.21.15.78:0/1530164923 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a9006aa90 con 0x7f5a9006e360 2023-10-14T11:59:49.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.525+0000 7f5a8f7fe700 1 --2- 172.21.15.78:0/1530164923 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a9006e360 0x7f5a90156650 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f5a900c4300 tx=0x7f5a80009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e0c13e49dacb224 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:49.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.525+0000 7f5a8effd700 1 -- 172.21.15.78:0/1530164923 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a80004230 con 0x7f5a9006e360 2023-10-14T11:59:49.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.525+0000 7f5a8effd700 1 -- 172.21.15.78:0/1530164923 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a80004390 con 0x7f5a9006e360 2023-10-14T11:59:49.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.525+0000 7f5a8effd700 1 -- 172.21.15.78:0/1530164923 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a800036a0 con 0x7f5a9006e360 2023-10-14T11:59:49.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.526+0000 7f5a9692d700 1 -- 172.21.15.78:0/1530164923 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a9006e360 msgr2=0x7f5a90156650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:49.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.526+0000 7f5a9692d700 1 --2- 172.21.15.78:0/1530164923 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a9006e360 0x7f5a90156650 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f5a900c4300 tx=0x7f5a80009bd0 comp rx=0 tx=0).stop 2023-10-14T11:59:49.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.526+0000 7f5a9692d700 1 -- 172.21.15.78:0/1530164923 shutdown_connections 2023-10-14T11:59:49.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.526+0000 7f5a9692d700 1 --2- 172.21.15.78:0/1530164923 >> v2:172.21.15.195:3300/0 conn(0x7f5a900abc90 0x7f5a900c1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:49.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.526+0000 7f5a9692d700 1 --2- 172.21.15.78:0/1530164923 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a9006e360 0x7f5a90156650 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:49.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.526+0000 7f5a9692d700 1 -- 172.21.15.78:0/1530164923 >> 172.21.15.78:0/1530164923 conn(0x7f5a900a8410 msgr2=0x7f5a900a8820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:49.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.526+0000 7f5a9692d700 1 -- 172.21.15.78:0/1530164923 shutdown_connections 2023-10-14T11:59:49.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.526+0000 7f5a9692d700 1 -- 172.21.15.78:0/1530164923 wait complete. 2023-10-14T11:59:49.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.527+0000 7f5a9692d700 1 Processor -- start 2023-10-14T11:59:49.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.527+0000 7f5a9692d700 1 -- start start 2023-10-14T11:59:49.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.528+0000 7f5a9692d700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5a9006e360 0x7f5a900c6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:49.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.528+0000 7f5a9692d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a900abc90 0x7f5a900bc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:49.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.528+0000 7f5a9692d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a90156b90 0x7f5a900bcc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:49.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.528+0000 7f5a9692d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a900ab220 con 0x7f5a900abc90 2023-10-14T11:59:49.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.528+0000 7f5a9692d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a901ea590 con 0x7f5a90156b90 2023-10-14T11:59:49.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.528+0000 7f5a9692d700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a901ea870 con 0x7f5a9006e360 2023-10-14T11:59:49.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.528+0000 7f5a8f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a900abc90 0x7f5a900bc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:49.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.528+0000 7f5a8f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a900abc90 0x7f5a900bc6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46830/0 (socket says 172.21.15.78:46830) 2023-10-14T11:59:49.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.528+0000 7f5a8f7fe700 1 -- 172.21.15.78:0/1039472853 learned_addr learned my addr 172.21.15.78:0/1039472853 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:49.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.528+0000 7f5a8ffff700 1 --2- 172.21.15.78:0/1039472853 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5a9006e360 0x7f5a900c6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:49.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.528+0000 7f5a94eca700 1 --2- 172.21.15.78:0/1039472853 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a90156b90 0x7f5a900bcc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:49.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.528+0000 7f5a8f7fe700 1 -- 172.21.15.78:0/1039472853 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5a9006e360 msgr2=0x7f5a900c6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:49.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.528+0000 7f5a8f7fe700 1 --2- 172.21.15.78:0/1039472853 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5a9006e360 0x7f5a900c6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:49.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.528+0000 7f5a8f7fe700 1 -- 172.21.15.78:0/1039472853 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a90156b90 msgr2=0x7f5a900bcc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:49.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.528+0000 7f5a8f7fe700 1 --2- 172.21.15.78:0/1039472853 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a90156b90 0x7f5a900bcc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:49.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.528+0000 7f5a8f7fe700 1 -- 172.21.15.78:0/1039472853 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a80009550 con 0x7f5a900abc90 2023-10-14T11:59:49.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.529+0000 7f5a8f7fe700 1 --2- 172.21.15.78:0/1039472853 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a900abc90 0x7f5a900bc6e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f5a80009130 tx=0x7f5a80004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:49.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.529+0000 7f5a8cff9700 1 -- 172.21.15.78:0/1039472853 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a80011630 con 0x7f5a900abc90 2023-10-14T11:59:49.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.529+0000 7f5a8cff9700 1 -- 172.21.15.78:0/1039472853 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a80011790 con 0x7f5a900abc90 2023-10-14T11:59:49.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.529+0000 7f5a9692d700 1 -- 172.21.15.78:0/1039472853 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a900a99b0 con 0x7f5a900abc90 2023-10-14T11:59:49.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.529+0000 7f5a8cff9700 1 -- 172.21.15.78:0/1039472853 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a80011b00 con 0x7f5a900abc90 2023-10-14T11:59:49.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.529+0000 7f5a9692d700 1 -- 172.21.15.78:0/1039472853 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a900bd160 con 0x7f5a900abc90 2023-10-14T11:59:49.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.530+0000 7f5a9692d700 1 -- 172.21.15.78:0/1039472853 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a900a9810 con 0x7f5a900abc90 2023-10-14T11:59:49.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.530+0000 7f5a8cff9700 1 -- 172.21.15.78:0/1039472853 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f5a80011c60 con 0x7f5a900abc90 2023-10-14T11:59:49.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.531+0000 7f5a8cff9700 1 --2- 172.21.15.78:0/1039472853 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f5a7006af30 0x7f5a7006d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:49.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.531+0000 7f5a8cff9700 1 -- 172.21.15.78:0/1039472853 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f5a8008ac90 con 0x7f5a900abc90 2023-10-14T11:59:49.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.531+0000 7f5a8ffff700 1 --2- 172.21.15.78:0/1039472853 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f5a7006af30 0x7f5a7006d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:49.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.531+0000 7f5a8ffff700 1 --2- 172.21.15.78:0/1039472853 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f5a7006af30 0x7f5a7006d3f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5a78006fd0 tx=0x7f5a78008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:49.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.535+0000 7f5a8cff9700 1 -- 172.21.15.78:0/1039472853 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5a900a9810 con 0x7f5a900abc90 2023-10-14T11:59:49.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.768+0000 7f5a9692d700 1 -- 172.21.15.78:0/1039472853 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a900a9810 con 0x7f5a900abc90 2023-10-14T11:59:49.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.768+0000 7f5a8cff9700 1 -- 172.21.15.78:0/1039472853 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4924 (secure 0 0 0) 0x7f5a900a9810 con 0x7f5a900abc90 2023-10-14T11:59:49.770 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:59:49.771 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4658,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6834/1492486786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1492486786},{"type":"v1","addr":"172.21.15.195:6834","nonce":1492486786}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5016,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/2991384297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2991384297},{"type":"v1","addr":"172.21.15.78:6836","nonce":2991384297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5019,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3246640037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3246640037},{"type":"v1","addr":"172.21.15.78:6837","nonce":3246640037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T11:59:01.562764+0000","modified":"2023-10-14T11:59:04.592935+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5031":{"gid":5031,"name":"d","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.195:6835/2130671508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6833","nonce":2130671508},{"type":"v1","addr":"172.21.15.195:6835","nonce":2130671508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"testcephfs1","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-14T11:59:49.772 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.770+0000 7f5a9692d700 1 -- 172.21.15.78:0/1039472853 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f5a7006af30 msgr2=0x7f5a7006d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:49.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.770+0000 7f5a9692d700 1 --2- 172.21.15.78:0/1039472853 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f5a7006af30 0x7f5a7006d3f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5a78006fd0 tx=0x7f5a78008040 comp rx=0 tx=0).stop 2023-10-14T11:59:49.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.770+0000 7f5a9692d700 1 -- 172.21.15.78:0/1039472853 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a900abc90 msgr2=0x7f5a900bc6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:49.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.770+0000 7f5a9692d700 1 --2- 172.21.15.78:0/1039472853 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a900abc90 0x7f5a900bc6e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f5a80009130 tx=0x7f5a80004750 comp rx=0 tx=0).stop 2023-10-14T11:59:49.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.770+0000 7f5a9692d700 1 -- 172.21.15.78:0/1039472853 shutdown_connections 2023-10-14T11:59:49.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.770+0000 7f5a9692d700 1 --2- 172.21.15.78:0/1039472853 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f5a7006af30 0x7f5a7006d3f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:49.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.770+0000 7f5a9692d700 1 --2- 172.21.15.78:0/1039472853 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a90156b90 0x7f5a900bcc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:49.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.770+0000 7f5a9692d700 1 --2- 172.21.15.78:0/1039472853 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5a9006e360 0x7f5a900c6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:49.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.770+0000 7f5a9692d700 1 --2- 172.21.15.78:0/1039472853 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a900abc90 0x7f5a900bc6e0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:49.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.770+0000 7f5a9692d700 1 -- 172.21.15.78:0/1039472853 >> 172.21.15.78:0/1039472853 conn(0x7f5a900a8410 msgr2=0x7f5a900a87f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:49.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.770+0000 7f5a9692d700 1 -- 172.21.15.78:0/1039472853 shutdown_connections 2023-10-14T11:59:49.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:49.770+0000 7f5a9692d700 1 -- 172.21.15.78:0/1039472853 wait complete. 2023-10-14T11:59:49.775 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 17 2023-10-14T11:59:50.187 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-14T11:59:01.562764+0000', 'modified': '2023-10-14T11:59:04.592935+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5031}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5031': {'gid': 5031, 'name': 'd', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.195:6835/2130671508', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6833', 'nonce': 2130671508}, {'type': 'v1', 'addr': '172.21.15.195:6835', 'nonce': 2130671508}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'testcephfs1', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-14T11:59:50.187 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-14T11:59:50.188 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-14T11:59:50.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.309+0000 7fe4d47e5700 1 Processor -- start 2023-10-14T11:59:50.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.309+0000 7fe4d47e5700 1 -- start start 2023-10-14T11:59:50.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.309+0000 7fe4d47e5700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe4cc0c4540 0x7fe4cc0c02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:50.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.309+0000 7fe4d47e5700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe4cc06b500 0x7fe4cc0c0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:50.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.309+0000 7fe4d47e5700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4cc06ebe0 con 0x7fe4cc13dbe0 2023-10-14T11:59:50.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.309+0000 7fe4d47e5700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4cc06d330 con 0x7fe4cc06b500 2023-10-14T11:59:50.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.309+0000 7fe4d47e5700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4cc0c0d60 con 0x7fe4cc0c4540 2023-10-14T11:59:50.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.310+0000 7fe4d2d82700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe4cc06b500 0x7fe4cc0c0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:50.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.310+0000 7fe4d2581700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fe4cc13dbe0 0x7fe4cc06a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51712/0 (socket says 172.21.15.78:51712) 2023-10-14T11:59:50.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.310+0000 7fe4d2581700 1 -- 172.21.15.78:0/92194452 learned_addr learned my addr 172.21.15.78:0/92194452 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:59:50.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.310+0000 7fe4d2d82700 1 -- 172.21.15.78:0/92194452 >> v2:172.21.15.195:3300/0 conn(0x7fe4cc0c4540 msgr2=0x7fe4cc0c02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:50.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.310+0000 7fe4d2d82700 1 --2- 172.21.15.78:0/92194452 >> v2:172.21.15.195:3300/0 conn(0x7fe4cc0c4540 0x7fe4cc0c02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.310+0000 7fe4d2d82700 1 -- 172.21.15.78:0/92194452 >> v1:172.21.15.78:6789/0 conn(0x7fe4cc13dbe0 legacy=0x7fe4cc06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:50.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.310+0000 7fe4d2d82700 1 -- 172.21.15.78:0/92194452 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4cc1558c0 con 0x7fe4cc06b500 2023-10-14T11:59:50.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.310+0000 7fe4d157f700 1 -- 172.21.15.78:0/92194452 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1999580697 0 0) 0x7fe4cc06ebe0 con 0x7fe4cc13dbe0 2023-10-14T11:59:50.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.310+0000 7fe4d2d82700 1 --2- 172.21.15.78:0/92194452 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4cc06b500 0x7fe4cc0c0820 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc13e360 tx=0x7fe4c400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9157f338050ea6d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:50.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.310+0000 7fe4d157f700 1 -- 172.21.15.78:0/92194452 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c400e070 con 0x7fe4cc06b500 2023-10-14T11:59:50.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.310+0000 7fe4d157f700 1 -- 172.21.15.78:0/92194452 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4c400bbd0 con 0x7fe4cc06b500 2023-10-14T11:59:50.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.310+0000 7fe4d157f700 1 -- 172.21.15.78:0/92194452 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c4012430 con 0x7fe4cc06b500 2023-10-14T11:59:50.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.311+0000 7fe4d47e5700 1 -- 172.21.15.78:0/92194452 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4cc06b500 msgr2=0x7fe4cc0c0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:50.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.311+0000 7fe4d47e5700 1 --2- 172.21.15.78:0/92194452 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4cc06b500 0x7fe4cc0c0820 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc13e360 tx=0x7fe4c400b330 comp rx=0 tx=0).stop 2023-10-14T11:59:50.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.311+0000 7fe4d47e5700 1 -- 172.21.15.78:0/92194452 shutdown_connections 2023-10-14T11:59:50.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.311+0000 7fe4d47e5700 1 --2- 172.21.15.78:0/92194452 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4cc06b500 0x7fe4cc0c0820 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.311+0000 7fe4d47e5700 1 --2- 172.21.15.78:0/92194452 >> v2:172.21.15.195:3300/0 conn(0x7fe4cc0c4540 0x7fe4cc0c02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.311+0000 7fe4d47e5700 1 -- 172.21.15.78:0/92194452 >> 172.21.15.78:0/92194452 conn(0x7fe4cc0c3a50 msgr2=0x7fe4cc0c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:50.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.311+0000 7fe4d47e5700 1 -- 172.21.15.78:0/92194452 shutdown_connections 2023-10-14T11:59:50.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.311+0000 7fe4d47e5700 1 -- 172.21.15.78:0/92194452 wait complete. 2023-10-14T11:59:50.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.312+0000 7fe4d47e5700 1 Processor -- start 2023-10-14T11:59:50.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.312+0000 7fe4d47e5700 1 -- start start 2023-10-14T11:59:50.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.313+0000 7fe4d47e5700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe4cc06b500 0x7fe4cc0c1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:50.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.313+0000 7fe4d47e5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe4cc0c4540 0x7fe4cc137030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:50.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.313+0000 7fe4d47e5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4cc13dbe0 0x7fe4cc0ab6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:50.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.313+0000 7fe4d47e5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4cc1579d0 con 0x7fe4cc13dbe0 2023-10-14T11:59:50.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.313+0000 7fe4d47e5700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4cc1e9c10 con 0x7fe4cc0c4540 2023-10-14T11:59:50.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.313+0000 7fe4d47e5700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4cc1379a0 con 0x7fe4cc06b500 2023-10-14T11:59:50.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.313+0000 7fe4d2d82700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4cc13dbe0 0x7fe4cc0ab6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:50.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.313+0000 7fe4d2d82700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4cc13dbe0 0x7fe4cc0ab6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46846/0 (socket says 172.21.15.78:46846) 2023-10-14T11:59:50.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.313+0000 7fe4d2d82700 1 -- 172.21.15.78:0/3881805677 learned_addr learned my addr 172.21.15.78:0/3881805677 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:50.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.313+0000 7fe4d2d82700 1 -- 172.21.15.78:0/3881805677 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe4cc06b500 msgr2=0x7fe4cc0c1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:50.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.313+0000 7fe4d2d82700 1 --2- 172.21.15.78:0/3881805677 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe4cc06b500 0x7fe4cc0c1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.313+0000 7fe4d2d82700 1 -- 172.21.15.78:0/3881805677 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe4cc0c4540 msgr2=0x7fe4cc137030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:50.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.313+0000 7fe4d2d82700 1 --2- 172.21.15.78:0/3881805677 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe4cc0c4540 0x7fe4cc137030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.313+0000 7fe4d2d82700 1 -- 172.21.15.78:0/3881805677 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c4009a70 con 0x7fe4cc13dbe0 2023-10-14T11:59:50.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.313+0000 7fe4d2d82700 1 --2- 172.21.15.78:0/3881805677 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4cc13dbe0 0x7fe4cc0ab6f0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc072fc0 tx=0x7fe4c4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:50.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.313+0000 7fe4c2ffd700 1 -- 172.21.15.78:0/3881805677 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c4012430 con 0x7fe4cc13dbe0 2023-10-14T11:59:50.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.313+0000 7fe4c2ffd700 1 -- 172.21.15.78:0/3881805677 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4c4020070 con 0x7fe4cc13dbe0 2023-10-14T11:59:50.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.313+0000 7fe4d47e5700 1 -- 172.21.15.78:0/3881805677 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4cc1e9e20 con 0x7fe4cc13dbe0 2023-10-14T11:59:50.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.313+0000 7fe4c2ffd700 1 -- 172.21.15.78:0/3881805677 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c4004280 con 0x7fe4cc13dbe0 2023-10-14T11:59:50.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.313+0000 7fe4d47e5700 1 -- 172.21.15.78:0/3881805677 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4cc1ea0c0 con 0x7fe4cc13dbe0 2023-10-14T11:59:50.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.315+0000 7fe4c2ffd700 1 -- 172.21.15.78:0/3881805677 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe4c40043e0 con 0x7fe4cc13dbe0 2023-10-14T11:59:50.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.315+0000 7fe4c2ffd700 1 --2- 172.21.15.78:0/3881805677 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe4b006af00 0x7fe4b006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:50.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.315+0000 7fe4d2581700 1 --2- 172.21.15.78:0/3881805677 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe4b006af00 0x7fe4b006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:50.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.315+0000 7fe4c2ffd700 1 -- 172.21.15.78:0/3881805677 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7fe4c4022070 con 0x7fe4cc13dbe0 2023-10-14T11:59:50.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.316+0000 7fe4d2581700 1 --2- 172.21.15.78:0/3881805677 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe4b006af00 0x7fe4b006d3c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc001610 tx=0x7fe4bc0079f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:50.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.317+0000 7fe4e09cfb80 1 Processor -- start 2023-10-14T11:59:50.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.317+0000 7fe4e09cfb80 1 -- start start 2023-10-14T11:59:50.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.318+0000 7fe4e09cfb80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55aacfb386b0 0x55aacfb76620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:50.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.318+0000 7fe4e09cfb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aacfb38a90 0x55aacfb76b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:50.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.318+0000 7fe4e09cfb80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55aacfb32c90 0x55aacfb568a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:50.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.318+0000 7fe4e09cfb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aacfaacc20 con 0x55aacfb38a90 2023-10-14T11:59:50.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.318+0000 7fe4e09cfb80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aacfa91c10 con 0x55aacfb32c90 2023-10-14T11:59:50.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.318+0000 7fe4e09cfb80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x55aacfa8f530 con 0x55aacfb386b0 2023-10-14T11:59:50.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.318+0000 7fe4d1d80700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aacfb38a90 0x55aacfb76b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:50.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.318+0000 7fe4d1d80700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aacfb38a90 0x55aacfb76b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46852/0 (socket says 172.21.15.78:46852) 2023-10-14T11:59:50.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.318+0000 7fe4d1d80700 1 -- 172.21.15.78:0/1438697995 learned_addr learned my addr 172.21.15.78:0/1438697995 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:50.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.318+0000 7fe4d2581700 1 --2- 172.21.15.78:0/1438697995 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55aacfb386b0 0x55aacfb76620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:50.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.318+0000 7fe4d1d80700 1 -- 172.21.15.78:0/1438697995 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55aacfb386b0 msgr2=0x55aacfb76620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:50.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.318+0000 7fe4d1d80700 1 --2- 172.21.15.78:0/1438697995 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55aacfb386b0 0x55aacfb76620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.318+0000 7fe4d1d80700 1 -- 172.21.15.78:0/1438697995 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55aacfb32c90 msgr2=0x55aacfb568a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:50.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.318+0000 7fe4d1d80700 1 --2- 172.21.15.78:0/1438697995 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55aacfb32c90 0x55aacfb568a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.318+0000 7fe4d1d80700 1 -- 172.21.15.78:0/1438697995 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aacfa483a0 con 0x55aacfb38a90 2023-10-14T11:59:50.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.319+0000 7fe4d1d80700 1 --2- 172.21.15.78:0/1438697995 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aacfb38a90 0x55aacfb76b60 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fe4c80098a0 tx=0x7fe4c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce3c833514a5705b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:50.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.319+0000 7fe4affff700 1 -- 172.21.15.78:0/1438697995 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c8011830 con 0x55aacfb38a90 2023-10-14T11:59:50.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.319+0000 7fe4affff700 1 -- 172.21.15.78:0/1438697995 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4c80119b0 con 0x55aacfb38a90 2023-10-14T11:59:50.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.319+0000 7fe4affff700 1 -- 172.21.15.78:0/1438697995 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c8011d60 con 0x55aacfb38a90 2023-10-14T11:59:50.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.319+0000 7fe4e09cfb80 1 -- 172.21.15.78:0/1438697995 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aacfb38a90 msgr2=0x55aacfb76b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:50.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.319+0000 7fe4e09cfb80 1 --2- 172.21.15.78:0/1438697995 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aacfb38a90 0x55aacfb76b60 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fe4c80098a0 tx=0x7fe4c8009bb0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.319+0000 7fe4e09cfb80 1 -- 172.21.15.78:0/1438697995 shutdown_connections 2023-10-14T11:59:50.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.319+0000 7fe4e09cfb80 1 --2- 172.21.15.78:0/1438697995 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55aacfb32c90 0x55aacfb568a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.319+0000 7fe4e09cfb80 1 --2- 172.21.15.78:0/1438697995 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55aacfb386b0 0x55aacfb76620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.319+0000 7fe4e09cfb80 1 --2- 172.21.15.78:0/1438697995 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aacfb38a90 0x55aacfb76b60 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.319+0000 7fe4e09cfb80 1 -- 172.21.15.78:0/1438697995 >> 172.21.15.78:0/1438697995 conn(0x55aacfa65d80 msgr2=0x55aacfb363a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:50.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.319+0000 7fe4e09cfb80 1 -- 172.21.15.78:0/1438697995 shutdown_connections 2023-10-14T11:59:50.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.319+0000 7fe4e09cfb80 1 -- 172.21.15.78:0/1438697995 wait complete. 2023-10-14T11:59:50.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.320+0000 7fe4e09cfb80 1 -- start start 2023-10-14T11:59:50.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.320+0000 7fe4e09cfb80 1 Processor -- start 2023-10-14T11:59:50.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.321+0000 7fe4e09cfb80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55aacfb32c90 0x55aacfb878d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:50.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.321+0000 7fe4e09cfb80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55aacfb386b0 0x55aacfb57640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:50.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.321+0000 7fe4e09cfb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aacfb38a90 0x55aacfb5d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:50.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.321+0000 7fe4e09cfb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aacfa8ae20 con 0x55aacfb38a90 2023-10-14T11:59:50.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.321+0000 7fe4e09cfb80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aacfa276c0 con 0x55aacfb386b0 2023-10-14T11:59:50.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.321+0000 7fe4e09cfb80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x55aacfa27360 con 0x55aacfb32c90 2023-10-14T11:59:50.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.321+0000 7fe4e09cfb80 10 client.0 _ll_register_callbacks cb 0x55aacf8847b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T11:59:50.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.321+0000 7fe4d2d82700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aacfb38a90 0x55aacfb5d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:50.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.321+0000 7fe4d2d82700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aacfb38a90 0x55aacfb5d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46860/0 (socket says 172.21.15.78:46860) 2023-10-14T11:59:50.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.321+0000 7fe4d2d82700 1 -- 172.21.15.78:0/3167213666 learned_addr learned my addr 172.21.15.78:0/3167213666 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:50.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.321+0000 7fe4d1d80700 1 --2- 172.21.15.78:0/3167213666 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55aacfb386b0 0x55aacfb57640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:50.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.321+0000 7fe4d2581700 1 --2- 172.21.15.78:0/3167213666 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55aacfb32c90 0x55aacfb878d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:50.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.321+0000 7fe4d2d82700 1 -- 172.21.15.78:0/3167213666 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55aacfb32c90 msgr2=0x55aacfb878d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:50.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.321+0000 7fe4d2d82700 1 --2- 172.21.15.78:0/3167213666 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55aacfb32c90 0x55aacfb878d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.321+0000 7fe4d2d82700 1 -- 172.21.15.78:0/3167213666 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55aacfb386b0 msgr2=0x55aacfb57640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:50.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.321+0000 7fe4d2d82700 1 --2- 172.21.15.78:0/3167213666 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55aacfb386b0 0x55aacfb57640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.321+0000 7fe4d2d82700 1 -- 172.21.15.78:0/3167213666 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4c80094a0 con 0x55aacfb38a90 2023-10-14T11:59:50.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.321+0000 7fe4d2581700 1 --2- 172.21.15.78:0/3167213666 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55aacfb32c90 0x55aacfb878d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:59:50.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.321+0000 7fe4d1d80700 1 --2- 172.21.15.78:0/3167213666 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55aacfb386b0 0x55aacfb57640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:59:50.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.322+0000 7fe4d2d82700 1 --2- 172.21.15.78:0/3167213666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aacfb38a90 0x55aacfb5d190 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4059f20 tx=0x7fe4c408b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=476410a4d7724ce0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:50.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.322+0000 7fe4ad7fa700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-14T11:59:50.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.322+0000 7fe4ad7fa700 1 -- 172.21.15.78:0/3167213666 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c4059910 con 0x55aacfb38a90 2023-10-14T11:59:50.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.322+0000 7fe4e09cfb80 1 -- 172.21.15.78:0/3167213666 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55aacfb2b5b0 con 0x55aacfb38a90 2023-10-14T11:59:50.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.322+0000 7fe4ad7fa700 1 -- 172.21.15.78:0/3167213666 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7fe4c4062520 con 0x55aacfb38a90 2023-10-14T11:59:50.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.322+0000 7fe4ad7fa700 1 -- 172.21.15.78:0/3167213666 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe4c405e170 con 0x55aacfb38a90 2023-10-14T11:59:50.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.322+0000 7fe4e09cfb80 10 client.5424 fetch_fsmap learned FSMap version 17 2023-10-14T11:59:50.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.322+0000 7fe4e09cfb80 1 -- 172.21.15.78:0/3167213666 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55aacf8b5d90 con 0x55aacfb38a90 2023-10-14T11:59:50.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.322+0000 7fe4ad7fa700 1 -- 172.21.15.78:0/3167213666 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55aacfb2b5b0 con 0x55aacfb38a90 2023-10-14T11:59:50.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.322+0000 7fe4ad7fa700 1 -- 172.21.15.78:0/3167213666 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 17) v1 ==== 3299+0+0 (secure 0 0 0) 0x7fe4c4062090 con 0x55aacfb38a90 2023-10-14T11:59:50.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.322+0000 7fe4e09cfb80 10 client.5424 fetch_fsmap finished waiting for FSMap version 17 2023-10-14T11:59:50.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.322+0000 7fe4e09cfb80 10 client.5424 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-14T11:59:50.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.322+0000 7fe4e09cfb80 20 client.5424 populate_metadata read hostname 'smithi078' 2023-10-14T11:59:50.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.322+0000 7fe4e09cfb80 1 --2- 172.21.15.78:0/3167213666 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x55aacfb83ee0 0x55aacfb842c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:50.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.322+0000 7fe4e09cfb80 4 client.5424 mds_command: new command op to 5031 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-14T11:59:50.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.322+0000 7fe4e09cfb80 1 -- 172.21.15.78:0/3167213666 --> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55aacfa83e00 con 0x55aacfb83ee0 2023-10-14T11:59:50.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.322+0000 7fe4d2581700 1 --2- 172.21.15.78:0/3167213666 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x55aacfb83ee0 0x55aacfb842c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:50.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.323+0000 7fe4ad7fa700 10 client.5424 ms_handle_connect on v2:172.21.15.195:6833/2130671508 2023-10-14T11:59:50.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.323+0000 7fe4d2581700 1 --2- 172.21.15.78:0/3167213666 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x55aacfb83ee0 0x55aacfb842c0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9a82dd21a137683 server_cookie=d08eccc15c119ada in_seq=0 out_seq=0 2023-10-14T11:59:50.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.323+0000 7fe4ad7fa700 1 -- 172.21.15.78:0/3167213666 <== mds.0 v2:172.21.15.195:6833/2130671508 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55aacfa83e00 con 0x55aacfb83ee0 2023-10-14T11:59:50.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.323+0000 7fe4ad7fa700 10 client.5424 handle_command_reply: tid=0 2023-10-14T11:59:50.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.323+0000 7fe4e09cfb80 1 client.5424 shutdown 2023-10-14T11:59:50.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.324+0000 7fe4e09cfb80 1 -- 172.21.15.78:0/3167213666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aacfb38a90 msgr2=0x55aacfb5d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:50.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.324+0000 7fe4e09cfb80 1 --2- 172.21.15.78:0/3167213666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aacfb38a90 0x55aacfb5d190 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4059f20 tx=0x7fe4c408b330 comp rx=0 tx=0).stop 2023-10-14T11:59:50.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.324+0000 7fe4e09cfb80 1 -- 172.21.15.78:0/3167213666 shutdown_connections 2023-10-14T11:59:50.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.324+0000 7fe4e09cfb80 1 --2- 172.21.15.78:0/3167213666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aacfb38a90 0x55aacfb5d190 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.324+0000 7fe4e09cfb80 1 --2- 172.21.15.78:0/3167213666 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55aacfb32c90 0x55aacfb878d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.324+0000 7fe4e09cfb80 1 --2- 172.21.15.78:0/3167213666 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x55aacfb83ee0 0x55aacfb842c0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.324+0000 7fe4e09cfb80 1 --2- 172.21.15.78:0/3167213666 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55aacfb386b0 0x55aacfb57640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.324+0000 7fe4ad7fa700 0 client.5424 ms_handle_reset on v2:172.21.15.195:6833/2130671508 2023-10-14T11:59:50.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.324+0000 7fe4e09cfb80 1 -- 172.21.15.78:0/3167213666 >> 172.21.15.78:0/3167213666 conn(0x55aacfa65d80 msgr2=0x55aacfb2d7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:50.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.324+0000 7fe4e09cfb80 1 -- 172.21.15.78:0/3167213666 shutdown_connections 2023-10-14T11:59:50.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.324+0000 7fe4e09cfb80 1 -- 172.21.15.78:0/3167213666 wait complete. 2023-10-14T11:59:50.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.324+0000 7fe4e09cfb80 20 client.5424 trim_cache size 0 max 16384 2023-10-14T11:59:50.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.341+0000 7fe4e09cfb80 1 Processor -- start 2023-10-14T11:59:50.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.341+0000 7fe4e09cfb80 1 -- start start 2023-10-14T11:59:50.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.342+0000 7fe4e09cfb80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55aacfb32c90 0x55aacfb71fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:50.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.342+0000 7fe4e09cfb80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55aacfb386b0 0x55aacfb810b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:50.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.342+0000 7fe4e09cfb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aacfb38a90 0x55aacfb77d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:50.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.342+0000 7fe4e09cfb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aacf90b8d0 con 0x55aacfb38a90 2023-10-14T11:59:50.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.342+0000 7fe4e09cfb80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aacfb882a0 con 0x55aacfb386b0 2023-10-14T11:59:50.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.342+0000 7fe4e09cfb80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x55aacfb32b30 con 0x55aacfb32c90 2023-10-14T11:59:50.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.342+0000 7fe4d2d82700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aacfb38a90 0x55aacfb77d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:50.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.342+0000 7fe4d2d82700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aacfb38a90 0x55aacfb77d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46866/0 (socket says 172.21.15.78:46866) 2023-10-14T11:59:50.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.342+0000 7fe4d2d82700 1 -- 172.21.15.78:0/2773746560 learned_addr learned my addr 172.21.15.78:0/2773746560 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:50.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.343+0000 7fe4d1d80700 1 --2- 172.21.15.78:0/2773746560 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55aacfb386b0 0x55aacfb810b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:50.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.343+0000 7fe4d2581700 1 --2- 172.21.15.78:0/2773746560 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55aacfb32c90 0x55aacfb71fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:50.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.343+0000 7fe4d2d82700 1 -- 172.21.15.78:0/2773746560 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55aacfb32c90 msgr2=0x55aacfb71fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:50.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.343+0000 7fe4d2d82700 1 --2- 172.21.15.78:0/2773746560 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55aacfb32c90 0x55aacfb71fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.343+0000 7fe4d2d82700 1 -- 172.21.15.78:0/2773746560 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55aacfb386b0 msgr2=0x55aacfb810b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:50.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.343+0000 7fe4d2d82700 1 --2- 172.21.15.78:0/2773746560 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55aacfb386b0 0x55aacfb810b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.343+0000 7fe4d2d82700 1 -- 172.21.15.78:0/2773746560 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aacf89ec10 con 0x55aacfb38a90 2023-10-14T11:59:50.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.343+0000 7fe4d2d82700 1 --2- 172.21.15.78:0/2773746560 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aacfb38a90 0x55aacfb77d20 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4058570 tx=0x7fe4c405c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34afa1351a44785 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:50.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.343+0000 7fe4c0ff9700 1 -- 172.21.15.78:0/2773746560 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c408b330 con 0x55aacfb38a90 2023-10-14T11:59:50.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.343+0000 7fe4c0ff9700 1 -- 172.21.15.78:0/2773746560 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4c405e170 con 0x55aacfb38a90 2023-10-14T11:59:50.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.343+0000 7fe4c0ff9700 1 -- 172.21.15.78:0/2773746560 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c4004900 con 0x55aacfb38a90 2023-10-14T11:59:50.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.343+0000 7fe4e09cfb80 1 -- 172.21.15.78:0/2773746560 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aacfb38a90 msgr2=0x55aacfb77d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:50.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.343+0000 7fe4e09cfb80 1 --2- 172.21.15.78:0/2773746560 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aacfb38a90 0x55aacfb77d20 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4058570 tx=0x7fe4c405c000 comp rx=0 tx=0).stop 2023-10-14T11:59:50.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.343+0000 7fe4e09cfb80 1 -- 172.21.15.78:0/2773746560 shutdown_connections 2023-10-14T11:59:50.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.343+0000 7fe4e09cfb80 1 --2- 172.21.15.78:0/2773746560 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aacfb38a90 0x55aacfb77d20 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.343+0000 7fe4e09cfb80 1 --2- 172.21.15.78:0/2773746560 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55aacfb32c90 0x55aacfb71fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.343+0000 7fe4e09cfb80 1 --2- 172.21.15.78:0/2773746560 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55aacfb386b0 0x55aacfb810b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.343+0000 7fe4e09cfb80 1 -- 172.21.15.78:0/2773746560 >> 172.21.15.78:0/2773746560 conn(0x55aacfa65d80 msgr2=0x55aacfb96b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:50.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.343+0000 7fe4e09cfb80 1 -- 172.21.15.78:0/2773746560 shutdown_connections 2023-10-14T11:59:50.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.344+0000 7fe4e09cfb80 1 -- 172.21.15.78:0/2773746560 wait complete. 2023-10-14T11:59:50.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.344+0000 7fe4e09cfb80 1 -- start start 2023-10-14T11:59:50.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.344+0000 7fe4e09cfb80 1 Processor -- start 2023-10-14T11:59:50.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.345+0000 7fe4e09cfb80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55aacfb32c90 0x55aacfb814b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:50.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.345+0000 7fe4e09cfb80 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55aacfb386b0 0x55aacfbdb590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:50.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.345+0000 7fe4e09cfb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aacfb38a90 0x55aacfbd8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:50.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.345+0000 7fe4e09cfb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aacfbdbff0 con 0x55aacfb38a90 2023-10-14T11:59:50.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.345+0000 7fe4e09cfb80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aacfb39f50 con 0x55aacfb32c90 2023-10-14T11:59:50.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.345+0000 7fe4e09cfb80 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x55aacfb78570 con 0x55aacfb386b0 2023-10-14T11:59:50.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.345+0000 7fe4e09cfb80 10 client.0 _ll_register_callbacks cb 0x7fe4c4059df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T11:59:50.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.345+0000 7fe4d2d82700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aacfb38a90 0x55aacfbd8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:50.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.345+0000 7fe4d2d82700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aacfb38a90 0x55aacfbd8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46872/0 (socket says 172.21.15.78:46872) 2023-10-14T11:59:50.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.345+0000 7fe4d2d82700 1 -- 172.21.15.78:0/1240581024 learned_addr learned my addr 172.21.15.78:0/1240581024 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:50.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.345+0000 7fe4d1d80700 1 --2- 172.21.15.78:0/1240581024 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55aacfb386b0 0x55aacfbdb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:50.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.345+0000 7fe4d2581700 1 --2- 172.21.15.78:0/1240581024 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55aacfb32c90 0x55aacfb814b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:50.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.345+0000 7fe4d2d82700 1 -- 172.21.15.78:0/1240581024 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55aacfb386b0 msgr2=0x55aacfbdb590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:50.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.345+0000 7fe4d2d82700 1 --2- 172.21.15.78:0/1240581024 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55aacfb386b0 0x55aacfbdb590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.345+0000 7fe4d2d82700 1 -- 172.21.15.78:0/1240581024 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55aacfb32c90 msgr2=0x55aacfb814b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:50.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.345+0000 7fe4d2d82700 1 --2- 172.21.15.78:0/1240581024 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55aacfb32c90 0x55aacfb814b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.345+0000 7fe4d2d82700 1 -- 172.21.15.78:0/1240581024 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aacf8b5d90 con 0x55aacfb38a90 2023-10-14T11:59:50.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.345+0000 7fe4d2d82700 1 --2- 172.21.15.78:0/1240581024 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aacfb38a90 0x55aacfbd8790 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4059f20 tx=0x7fe4c4012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ab58e11c02fbb42 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:50.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.345+0000 7fe4ad7fa700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-14T11:59:50.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.345+0000 7fe4ad7fa700 1 -- 172.21.15.78:0/1240581024 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c4004890 con 0x55aacfb38a90 2023-10-14T11:59:50.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.346+0000 7fe4ad7fa700 1 -- 172.21.15.78:0/1240581024 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7fe4c4062520 con 0x55aacfb38a90 2023-10-14T11:59:50.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.346+0000 7fe4e09cfb80 1 -- 172.21.15.78:0/1240581024 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe4c80119b0 con 0x55aacfb38a90 2023-10-14T11:59:50.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.346+0000 7fe4ad7fa700 1 -- 172.21.15.78:0/1240581024 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe4c4003e10 con 0x55aacfb38a90 2023-10-14T11:59:50.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.346+0000 7fe4e09cfb80 10 client.5430 fetch_fsmap learned FSMap version 17 2023-10-14T11:59:50.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.346+0000 7fe4e09cfb80 1 -- 172.21.15.78:0/1240581024 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55aacf8dc200 con 0x55aacfb38a90 2023-10-14T11:59:50.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.346+0000 7fe4ad7fa700 1 -- 172.21.15.78:0/1240581024 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4c80119b0 con 0x55aacfb38a90 2023-10-14T11:59:50.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.346+0000 7fe4ad7fa700 1 -- 172.21.15.78:0/1240581024 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 17) v1 ==== 3299+0+0 (secure 0 0 0) 0x7fe4c4080070 con 0x55aacfb38a90 2023-10-14T11:59:50.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.346+0000 7fe4e09cfb80 10 client.5430 fetch_fsmap finished waiting for FSMap version 17 2023-10-14T11:59:50.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.346+0000 7fe4e09cfb80 10 client.5430 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-14T11:59:50.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.346+0000 7fe4e09cfb80 20 client.5430 populate_metadata read hostname 'smithi078' 2023-10-14T11:59:50.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.346+0000 7fe4e09cfb80 1 --2- 172.21.15.78:0/1240581024 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x55aacfbd8cd0 0x55aacfb72a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:50.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.346+0000 7fe4e09cfb80 4 client.5430 mds_command: new command op to 5031 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-14T11:59:50.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.346+0000 7fe4e09cfb80 1 -- 172.21.15.78:0/1240581024 --> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55aacfa83b20 con 0x55aacfbd8cd0 2023-10-14T11:59:50.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.346+0000 7fe4d2581700 1 --2- 172.21.15.78:0/1240581024 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x55aacfbd8cd0 0x55aacfb72a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:50.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.347+0000 7fe4ad7fa700 10 client.5430 ms_handle_connect on v2:172.21.15.195:6833/2130671508 2023-10-14T11:59:50.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.347+0000 7fe4d2581700 1 --2- 172.21.15.78:0/1240581024 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x55aacfbd8cd0 0x55aacfb72a30 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3dea7380df15797 server_cookie=aa1cd19d6b9aebb6 in_seq=0 out_seq=0 2023-10-14T11:59:50.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.347+0000 7fe4ad7fa700 1 -- 172.21.15.78:0/1240581024 <== mds.0 v2:172.21.15.195:6833/2130671508 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+336 (crc 0 0 0) 0x55aacfa83b20 con 0x55aacfbd8cd0 2023-10-14T11:59:50.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.347+0000 7fe4ad7fa700 10 client.5430 handle_command_reply: tid=0 2023-10-14T11:59:50.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.347+0000 7fe4e09cfb80 1 client.5430 shutdown 2023-10-14T11:59:50.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.347+0000 7fe4e09cfb80 1 -- 172.21.15.78:0/1240581024 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aacfb38a90 msgr2=0x55aacfbd8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:50.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.347+0000 7fe4e09cfb80 1 --2- 172.21.15.78:0/1240581024 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aacfb38a90 0x55aacfbd8790 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4059f20 tx=0x7fe4c4012000 comp rx=0 tx=0).stop 2023-10-14T11:59:50.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.347+0000 7fe4e09cfb80 1 -- 172.21.15.78:0/1240581024 shutdown_connections 2023-10-14T11:59:50.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.347+0000 7fe4e09cfb80 1 --2- 172.21.15.78:0/1240581024 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aacfb38a90 0x55aacfbd8790 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.347+0000 7fe4e09cfb80 1 --2- 172.21.15.78:0/1240581024 >> [v2:172.21.15.195:6833/2130671508,v1:172.21.15.195:6835/2130671508] conn(0x55aacfbd8cd0 0x55aacfb72a30 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.347+0000 7fe4e09cfb80 1 --2- 172.21.15.78:0/1240581024 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55aacfb32c90 0x55aacfb814b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.347+0000 7fe4e09cfb80 1 --2- 172.21.15.78:0/1240581024 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55aacfb386b0 0x55aacfbdb590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.347+0000 7fe4ad7fa700 0 client.5430 ms_handle_reset on v2:172.21.15.195:6833/2130671508 2023-10-14T11:59:50.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.347+0000 7fe4e09cfb80 1 -- 172.21.15.78:0/1240581024 >> 172.21.15.78:0/1240581024 conn(0x55aacfa65d80 msgr2=0x55aacfb363a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:50.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.347+0000 7fe4e09cfb80 1 -- 172.21.15.78:0/1240581024 shutdown_connections 2023-10-14T11:59:50.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.347+0000 7fe4e09cfb80 1 -- 172.21.15.78:0/1240581024 wait complete. 2023-10-14T11:59:50.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.347+0000 7fe4e09cfb80 20 client.5430 trim_cache size 0 max 16384 2023-10-14T11:59:50.378 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-14T11:59:50.378 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "4c5dc134-3092-406a-a1e1-13e6d9e926e8", 2023-10-14T11:59:50.378 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2023-10-14T11:59:50.379 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 5031, 2023-10-14T11:59:50.379 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2023-10-14T11:59:50.379 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2023-10-14T11:59:50.379 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "testcephfs1", 2023-10-14T11:59:50.379 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 48.162924371000003, 2023-10-14T11:59:50.380 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 17, 2023-10-14T11:59:50.380 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 32, 2023-10-14T11:59:50.380 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 32, 2023-10-14T11:59:50.380 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 56.053747805999997 2023-10-14T11:59:50.380 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-14T11:59:50.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.347+0000 7fe4d47e5700 1 -- 172.21.15.78:0/3881805677 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe4b006af00 msgr2=0x7fe4b006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:50.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.347+0000 7fe4d47e5700 1 --2- 172.21.15.78:0/3881805677 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe4b006af00 0x7fe4b006d3c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc001610 tx=0x7fe4bc0079f0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.347+0000 7fe4d47e5700 1 -- 172.21.15.78:0/3881805677 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4cc13dbe0 msgr2=0x7fe4cc0ab6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:50.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.347+0000 7fe4d47e5700 1 --2- 172.21.15.78:0/3881805677 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4cc13dbe0 0x7fe4cc0ab6f0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc072fc0 tx=0x7fe4c4004ad0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.348+0000 7fe4d47e5700 1 -- 172.21.15.78:0/3881805677 shutdown_connections 2023-10-14T11:59:50.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.348+0000 7fe4d47e5700 1 --2- 172.21.15.78:0/3881805677 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4cc13dbe0 0x7fe4cc0ab6f0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.348+0000 7fe4d47e5700 1 --2- 172.21.15.78:0/3881805677 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe4cc06b500 0x7fe4cc0c1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.348+0000 7fe4d47e5700 1 --2- 172.21.15.78:0/3881805677 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fe4b006af00 0x7fe4b006d3c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.348+0000 7fe4d47e5700 1 --2- 172.21.15.78:0/3881805677 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe4cc0c4540 0x7fe4cc137030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.348+0000 7fe4d47e5700 1 -- 172.21.15.78:0/3881805677 >> 172.21.15.78:0/3881805677 conn(0x7fe4cc0c3a50 msgr2=0x7fe4cc13b560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:50.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.348+0000 7fe4d47e5700 1 -- 172.21.15.78:0/3881805677 shutdown_connections 2023-10-14T11:59:50.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.348+0000 7fe4d47e5700 1 -- 172.21.15.78:0/3881805677 wait complete. 2023-10-14T11:59:50.384 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 17, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-14T11:59:01.562764+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 17, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "testcephfs1", "in": [ 0 ], "info": { "gid_5031": { "addr": "172.21.15.195:6835/2130671508", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6833", "nonce": 2130671508, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2130671508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5031, "incarnation": 13, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2023-10-14T11:59:04.592935+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5031 } } } ], "standbys": [ { "addr": "172.21.15.195:6834/1492486786", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6832", "nonce": 1492486786, "type": "v2" }, { "addr": "172.21.15.195:6834", "nonce": 1492486786, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4658, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6836/2991384297", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6834", "nonce": 2991384297, "type": "v2" }, { "addr": "172.21.15.78:6836", "nonce": 2991384297, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5016, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6837/3246640037", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6835", "nonce": 3246640037, "type": "v2" }, { "addr": "172.21.15.78:6837", "nonce": 3246640037, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5019, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-14T11:59:50.386 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 2, 'name': 'b', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2023-10-14T11:59:50.386 INFO:tasks.cephfs.test_failover:current = [{'gid': 5019, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6837/3246640037', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 3246640037}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 3246640037}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 4658, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.195:6834/1492486786', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6832', 'nonce': 1492486786}, {'type': 'v1', 'addr': '172.21.15.195:6834', 'nonce': 1492486786}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 5016, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6836/2991384297', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 2991384297}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 2991384297}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 5031, 'name': 'd', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.195:6835/2130671508', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6833', 'nonce': 2130671508}, {'type': 'v1', 'addr': '172.21.15.195:6835', 'nonce': 2130671508}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2023-10-14T11:59:50.387 DEBUG:tasks.cephfs.test_failover:2 != -1 2023-10-14T11:59:50.387 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-14T11:59:50.388 DEBUG:teuthology.orchestra.run.smithi078:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T11:59:50.436 DEBUG:teuthology.orchestra.run.smithi195:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T11:59:50.480 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-14T11:59:50.481 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-14T11:59:50.482 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T11:59:50.499 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17259 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 2023-10-14T11:59:50.500 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-14T11:59:50.500 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T11:59:50.540 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15735 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 2023-10-14T11:59:50.541 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T11:59:50.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.659+0000 7f21e0cff700 1 Processor -- start 2023-10-14T11:59:50.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.659+0000 7f21e0cff700 1 -- start start 2023-10-14T11:59:50.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.660+0000 7f21e0cff700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f21dc06ab40 0x7f21dc06f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:50.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.660+0000 7f21e0cff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f21dc1554e0 0x7f21dc06af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:50.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.660+0000 7f21e0cff700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21dc0c49f0 con 0x7f21dc0c0350 2023-10-14T11:59:50.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.660+0000 7f21e0cff700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21dc0726a0 con 0x7f21dc06ab40 2023-10-14T11:59:50.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.660+0000 7f21e0cff700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21dc072980 con 0x7f21dc1554e0 2023-10-14T11:59:50.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.660+0000 7f21da59c700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f21dc06ab40 0x7f21dc06f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:50.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.660+0000 7f21da59c700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f21dc06ab40 0x7f21dc06f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46876/0 (socket says 172.21.15.78:46876) 2023-10-14T11:59:50.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.660+0000 7f21da59c700 1 -- 172.21.15.78:0/2417710214 learned_addr learned my addr 172.21.15.78:0/2417710214 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:50.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.660+0000 7f21d9d9b700 1 --2- 172.21.15.78:0/2417710214 >> v2:172.21.15.195:3300/0 conn(0x7f21dc1554e0 0x7f21dc06af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:50.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.660+0000 7f21da59c700 1 -- 172.21.15.78:0/2417710214 >> v2:172.21.15.195:3300/0 conn(0x7f21dc1554e0 msgr2=0x7f21dc06af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:50.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.660+0000 7f21da59c700 1 --2- 172.21.15.78:0/2417710214 >> v2:172.21.15.195:3300/0 conn(0x7f21dc1554e0 0x7f21dc06af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.660+0000 7f21da59c700 1 -- 172.21.15.78:0/2417710214 >> v1:172.21.15.195:6789/0 conn(0x7f21dc0c0350 legacy=0x7f21dc13d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:50.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.660+0000 7f21da59c700 1 -- 172.21.15.78:0/2417710214 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21dc154a00 con 0x7f21dc06ab40 2023-10-14T11:59:50.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.660+0000 7f21d9d9b700 1 --2- 172.21.15.78:0/2417710214 >> v2:172.21.15.195:3300/0 conn(0x7f21dc1554e0 0x7f21dc06af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:59:50.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.660+0000 7f21da59c700 1 --2- 172.21.15.78:0/2417710214 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21dc06ab40 0x7f21dc06f1e0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f21dc06ff00 tx=0x7f21d0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1018c21fd27ecf6a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:50.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.660+0000 7f21d959a700 1 -- 172.21.15.78:0/2417710214 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d0004230 con 0x7f21dc06ab40 2023-10-14T11:59:50.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.661+0000 7f21d959a700 1 -- 172.21.15.78:0/2417710214 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d0004390 con 0x7f21dc06ab40 2023-10-14T11:59:50.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.661+0000 7f21d959a700 1 -- 172.21.15.78:0/2417710214 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d00036a0 con 0x7f21dc06ab40 2023-10-14T11:59:50.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.661+0000 7f21e0cff700 1 -- 172.21.15.78:0/2417710214 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21dc06ab40 msgr2=0x7f21dc06f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:50.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.661+0000 7f21e0cff700 1 --2- 172.21.15.78:0/2417710214 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21dc06ab40 0x7f21dc06f1e0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f21dc06ff00 tx=0x7f21d0009bd0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.661+0000 7f21e0cff700 1 -- 172.21.15.78:0/2417710214 shutdown_connections 2023-10-14T11:59:50.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.661+0000 7f21e0cff700 1 --2- 172.21.15.78:0/2417710214 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21dc06ab40 0x7f21dc06f1e0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.661+0000 7f21e0cff700 1 --2- 172.21.15.78:0/2417710214 >> v2:172.21.15.195:3300/0 conn(0x7f21dc1554e0 0x7f21dc06af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.661+0000 7f21e0cff700 1 -- 172.21.15.78:0/2417710214 >> 172.21.15.78:0/2417710214 conn(0x7f21dc0c3ac0 msgr2=0x7f21dc0c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:50.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.661+0000 7f21e0cff700 1 -- 172.21.15.78:0/2417710214 shutdown_connections 2023-10-14T11:59:50.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.661+0000 7f21e0cff700 1 -- 172.21.15.78:0/2417710214 wait complete. 2023-10-14T11:59:50.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.662+0000 7f21e0cff700 1 Processor -- start 2023-10-14T11:59:50.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.663+0000 7f21e0cff700 1 -- start start 2023-10-14T11:59:50.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.663+0000 7f21e0cff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21dc06ab40 0x7f21dc153b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:50.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.663+0000 7f21e0cff700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f21dc0c0350 0x7f21dc156b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:50.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.663+0000 7f21e0cff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21dc1554e0 0x7f21dc157070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:50.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.663+0000 7f21e0cff700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc1ea430 con 0x7f21dc1554e0 2023-10-14T11:59:50.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.663+0000 7f21e0cff700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc1575b0 con 0x7f21dc06ab40 2023-10-14T11:59:50.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.663+0000 7f21e0cff700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc158db0 con 0x7f21dc0c0350 2023-10-14T11:59:50.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.663+0000 7f21dad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21dc1554e0 0x7f21dc157070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:50.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.663+0000 7f21dad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21dc1554e0 0x7f21dc157070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46884/0 (socket says 172.21.15.78:46884) 2023-10-14T11:59:50.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.663+0000 7f21dad9d700 1 -- 172.21.15.78:0/3629377934 learned_addr learned my addr 172.21.15.78:0/3629377934 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:50.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.663+0000 7f21da59c700 1 --2- 172.21.15.78:0/3629377934 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21dc06ab40 0x7f21dc153b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:50.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.663+0000 7f21d9d9b700 1 --2- 172.21.15.78:0/3629377934 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f21dc0c0350 0x7f21dc156b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:50.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.664+0000 7f21dad9d700 1 -- 172.21.15.78:0/3629377934 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f21dc0c0350 msgr2=0x7f21dc156b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:50.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.664+0000 7f21dad9d700 1 --2- 172.21.15.78:0/3629377934 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f21dc0c0350 0x7f21dc156b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.664+0000 7f21dad9d700 1 -- 172.21.15.78:0/3629377934 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21dc06ab40 msgr2=0x7f21dc153b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:50.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.664+0000 7f21dad9d700 1 --2- 172.21.15.78:0/3629377934 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21dc06ab40 0x7f21dc153b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.664+0000 7f21dad9d700 1 -- 172.21.15.78:0/3629377934 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d0009550 con 0x7f21dc1554e0 2023-10-14T11:59:50.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.664+0000 7f21dad9d700 1 --2- 172.21.15.78:0/3629377934 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21dc1554e0 0x7f21dc157070 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f21d400c7a0 tx=0x7f21d400cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:50.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.664+0000 7f21caffd700 1 -- 172.21.15.78:0/3629377934 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d4003e10 con 0x7f21dc1554e0 2023-10-14T11:59:50.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.664+0000 7f21caffd700 1 -- 172.21.15.78:0/3629377934 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d4005070 con 0x7f21dc1554e0 2023-10-14T11:59:50.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.664+0000 7f21e0cff700 1 -- 172.21.15.78:0/3629377934 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21dc1ea850 con 0x7f21dc1554e0 2023-10-14T11:59:50.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.664+0000 7f21caffd700 1 -- 172.21.15.78:0/3629377934 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d4005420 con 0x7f21dc1554e0 2023-10-14T11:59:50.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.664+0000 7f21e0cff700 1 -- 172.21.15.78:0/3629377934 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21dc1e9d70 con 0x7f21dc1554e0 2023-10-14T11:59:50.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.665+0000 7f21e0cff700 1 -- 172.21.15.78:0/3629377934 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21dc074b80 con 0x7f21dc1554e0 2023-10-14T11:59:50.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.665+0000 7f21caffd700 1 -- 172.21.15.78:0/3629377934 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f21d4005580 con 0x7f21dc1554e0 2023-10-14T11:59:50.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.666+0000 7f21caffd700 1 --2- 172.21.15.78:0/3629377934 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f21bc06af00 0x7f21bc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:50.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.666+0000 7f21caffd700 1 -- 172.21.15.78:0/3629377934 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f21d4088d30 con 0x7f21dc1554e0 2023-10-14T11:59:50.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.666+0000 7f21da59c700 1 --2- 172.21.15.78:0/3629377934 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f21bc06af00 0x7f21bc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:50.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.666+0000 7f21da59c700 1 --2- 172.21.15.78:0/3629377934 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f21bc06af00 0x7f21bc06d3c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009130 tx=0x7f21d0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:50.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.670+0000 7f21caffd700 1 -- 172.21.15.78:0/3629377934 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f21d4053db0 con 0x7f21dc1554e0 2023-10-14T11:59:50.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.904+0000 7f21e0cff700 1 -- 172.21.15.78:0/3629377934 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21dc0c6af0 con 0x7f21dc1554e0 2023-10-14T11:59:50.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.904+0000 7f21caffd700 1 -- 172.21.15.78:0/3629377934 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4924 (secure 0 0 0) 0x7f21d4017070 con 0x7f21dc1554e0 2023-10-14T11:59:50.907 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:59:50.907 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4658,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6834/1492486786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1492486786},{"type":"v1","addr":"172.21.15.195:6834","nonce":1492486786}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5016,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/2991384297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2991384297},{"type":"v1","addr":"172.21.15.78:6836","nonce":2991384297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5019,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3246640037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3246640037},{"type":"v1","addr":"172.21.15.78:6837","nonce":3246640037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T11:59:01.562764+0000","modified":"2023-10-14T11:59:04.592935+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5031":{"gid":5031,"name":"d","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.195:6835/2130671508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6833","nonce":2130671508},{"type":"v1","addr":"172.21.15.195:6835","nonce":2130671508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"testcephfs1","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-14T11:59:50.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.906+0000 7f21e0cff700 1 -- 172.21.15.78:0/3629377934 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f21bc06af00 msgr2=0x7f21bc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:50.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.906+0000 7f21e0cff700 1 --2- 172.21.15.78:0/3629377934 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f21bc06af00 0x7f21bc06d3c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009130 tx=0x7f21d0008030 comp rx=0 tx=0).stop 2023-10-14T11:59:50.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.906+0000 7f21e0cff700 1 -- 172.21.15.78:0/3629377934 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21dc1554e0 msgr2=0x7f21dc157070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:50.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.906+0000 7f21e0cff700 1 --2- 172.21.15.78:0/3629377934 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21dc1554e0 0x7f21dc157070 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f21d400c7a0 tx=0x7f21d400cb60 comp rx=0 tx=0).stop 2023-10-14T11:59:50.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.907+0000 7f21e0cff700 1 -- 172.21.15.78:0/3629377934 shutdown_connections 2023-10-14T11:59:50.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.907+0000 7f21e0cff700 1 --2- 172.21.15.78:0/3629377934 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21dc1554e0 0x7f21dc157070 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.907+0000 7f21e0cff700 1 --2- 172.21.15.78:0/3629377934 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f21bc06af00 0x7f21bc06d3c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.907+0000 7f21e0cff700 1 --2- 172.21.15.78:0/3629377934 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21dc06ab40 0x7f21dc153b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.907+0000 7f21e0cff700 1 --2- 172.21.15.78:0/3629377934 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f21dc0c0350 0x7f21dc156b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:50.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.907+0000 7f21e0cff700 1 -- 172.21.15.78:0/3629377934 >> 172.21.15.78:0/3629377934 conn(0x7f21dc0c3ac0 msgr2=0x7f21dc0c2110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:50.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.907+0000 7f21e0cff700 1 -- 172.21.15.78:0/3629377934 shutdown_connections 2023-10-14T11:59:50.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:50.907+0000 7f21e0cff700 1 -- 172.21.15.78:0/3629377934 wait complete. 2023-10-14T11:59:50.912 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 17 2023-10-14T11:59:50.921 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T11:59:51.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.039+0000 7f0b7965c700 1 Processor -- start 2023-10-14T11:59:51.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.039+0000 7f0b7965c700 1 -- start start 2023-10-14T11:59:51.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.040+0000 7f0b7965c700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0b7414b7e0 0x7f0b740c5150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:51.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.040+0000 7f0b7965c700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0b7406cd00 0x7f0b7406b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:51.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.040+0000 7f0b7965c700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b740c9970 con 0x7f0b74154b90 2023-10-14T11:59:51.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.040+0000 7f0b7965c700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b740c4e40 con 0x7f0b7414b7e0 2023-10-14T11:59:51.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.040+0000 7f0b7965c700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b740c81c0 con 0x7f0b7406cd00 2023-10-14T11:59:51.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.040+0000 7f0b72ffd700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0b7414b7e0 0x7f0b740c5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:51.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.040+0000 7f0b727fc700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f0b74154b90 0x7f0b740c9260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51764/0 (socket says 172.21.15.78:51764) 2023-10-14T11:59:51.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.040+0000 7f0b72ffd700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0b7414b7e0 0x7f0b740c5150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46888/0 (socket says 172.21.15.78:46888) 2023-10-14T11:59:51.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.040+0000 7f0b727fc700 1 -- 172.21.15.78:0/3338158184 learned_addr learned my addr 172.21.15.78:0/3338158184 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:59:51.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.041+0000 7f0b737fe700 1 --2- 172.21.15.78:0/3338158184 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0b7406cd00 0x7f0b7406b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:51.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.041+0000 7f0b72ffd700 1 -- 172.21.15.78:0/3338158184 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0b7406cd00 msgr2=0x7f0b7406b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:51.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.041+0000 7f0b72ffd700 1 --2- 172.21.15.78:0/3338158184 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0b7406cd00 0x7f0b7406b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:51.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.041+0000 7f0b72ffd700 1 -- 172.21.15.78:0/3338158184 >> v1:172.21.15.78:6789/0 conn(0x7f0b74154b90 legacy=0x7f0b740c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:51.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.041+0000 7f0b72ffd700 1 -- 172.21.15.78:0/3338158184 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b740c84a0 con 0x7f0b7414b7e0 2023-10-14T11:59:51.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.041+0000 7f0b71ffb700 1 -- 172.21.15.78:0/3338158184 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4289285655 0 0) 0x7f0b740c9970 con 0x7f0b74154b90 2023-10-14T11:59:51.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.041+0000 7f0b737fe700 1 --2- 172.21.15.78:0/3338158184 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0b7406cd00 0x7f0b7406b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:59:51.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.041+0000 7f0b72ffd700 1 --2- 172.21.15.78:0/3338158184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b7414b7e0 0x7f0b740c5150 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f0b68009840 tx=0x7f0b68009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f292197a6148d7dd server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:51.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.041+0000 7f0b71ffb700 1 -- 172.21.15.78:0/3338158184 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b68011830 con 0x7f0b7414b7e0 2023-10-14T11:59:51.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.041+0000 7f0b71ffb700 1 -- 172.21.15.78:0/3338158184 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b68011990 con 0x7f0b7414b7e0 2023-10-14T11:59:51.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.041+0000 7f0b71ffb700 1 -- 172.21.15.78:0/3338158184 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b68011d20 con 0x7f0b7414b7e0 2023-10-14T11:59:51.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.041+0000 7f0b7965c700 1 -- 172.21.15.78:0/3338158184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b7414b7e0 msgr2=0x7f0b740c5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:51.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.042+0000 7f0b7965c700 1 --2- 172.21.15.78:0/3338158184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b7414b7e0 0x7f0b740c5150 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f0b68009840 tx=0x7f0b68009b50 comp rx=0 tx=0).stop 2023-10-14T11:59:51.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.042+0000 7f0b7965c700 1 -- 172.21.15.78:0/3338158184 shutdown_connections 2023-10-14T11:59:51.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.042+0000 7f0b7965c700 1 --2- 172.21.15.78:0/3338158184 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0b7406cd00 0x7f0b7406b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:51.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.042+0000 7f0b7965c700 1 --2- 172.21.15.78:0/3338158184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b7414b7e0 0x7f0b740c5150 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:51.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.042+0000 7f0b7965c700 1 -- 172.21.15.78:0/3338158184 >> 172.21.15.78:0/3338158184 conn(0x7f0b740ca2c0 msgr2=0x7f0b740ca6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:51.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.042+0000 7f0b7965c700 1 -- 172.21.15.78:0/3338158184 shutdown_connections 2023-10-14T11:59:51.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.042+0000 7f0b7965c700 1 -- 172.21.15.78:0/3338158184 wait complete. 2023-10-14T11:59:51.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.043+0000 7f0b7965c700 1 Processor -- start 2023-10-14T11:59:51.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.043+0000 7f0b7965c700 1 -- start start 2023-10-14T11:59:51.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.043+0000 7f0b7965c700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0b7406cd00 0x7f0b74147c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:51.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.044+0000 7f0b7965c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b7414b7e0 0x7f0b740c0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:51.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.044+0000 7f0b7965c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b74154b90 0x7f0b740c0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:51.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.044+0000 7f0b7965c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b741446c0 con 0x7f0b74154b90 2023-10-14T11:59:51.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.044+0000 7f0b7965c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b740bfb10 con 0x7f0b7414b7e0 2023-10-14T11:59:51.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.044+0000 7f0b7965c700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b740c3820 con 0x7f0b7406cd00 2023-10-14T11:59:51.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.044+0000 7f0b737fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b74154b90 0x7f0b740c0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:51.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.044+0000 7f0b737fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b74154b90 0x7f0b740c0770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46898/0 (socket says 172.21.15.78:46898) 2023-10-14T11:59:51.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.044+0000 7f0b737fe700 1 -- 172.21.15.78:0/1343627009 learned_addr learned my addr 172.21.15.78:0/1343627009 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:51.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.044+0000 7f0b72ffd700 1 --2- 172.21.15.78:0/1343627009 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0b7406cd00 0x7f0b74147c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:51.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.044+0000 7f0b727fc700 1 --2- 172.21.15.78:0/1343627009 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b7414b7e0 0x7f0b740c0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:51.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.044+0000 7f0b737fe700 1 -- 172.21.15.78:0/1343627009 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0b7406cd00 msgr2=0x7f0b74147c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:51.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.044+0000 7f0b737fe700 1 --2- 172.21.15.78:0/1343627009 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0b7406cd00 0x7f0b74147c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:51.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.044+0000 7f0b737fe700 1 -- 172.21.15.78:0/1343627009 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b7414b7e0 msgr2=0x7f0b740c0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:51.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.044+0000 7f0b737fe700 1 --2- 172.21.15.78:0/1343627009 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b7414b7e0 0x7f0b740c0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:51.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.044+0000 7f0b737fe700 1 -- 172.21.15.78:0/1343627009 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b680094a0 con 0x7f0b74154b90 2023-10-14T11:59:51.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.044+0000 7f0b727fc700 1 --2- 172.21.15.78:0/1343627009 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b7414b7e0 0x7f0b740c0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:59:51.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.044+0000 7f0b737fe700 1 --2- 172.21.15.78:0/1343627009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b74154b90 0x7f0b740c0770 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c007ef0 tx=0x7f0b6c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:51.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.044+0000 7f0b5b7fe700 1 -- 172.21.15.78:0/1343627009 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c00de30 con 0x7f0b74154b90 2023-10-14T11:59:51.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.044+0000 7f0b5b7fe700 1 -- 172.21.15.78:0/1343627009 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b6c00f040 con 0x7f0b74154b90 2023-10-14T11:59:51.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.044+0000 7f0b7965c700 1 -- 172.21.15.78:0/1343627009 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b740c41a0 con 0x7f0b74154b90 2023-10-14T11:59:51.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.044+0000 7f0b5b7fe700 1 -- 172.21.15.78:0/1343627009 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c003c60 con 0x7f0b74154b90 2023-10-14T11:59:51.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.044+0000 7f0b7965c700 1 -- 172.21.15.78:0/1343627009 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b740c3ec0 con 0x7f0b74154b90 2023-10-14T11:59:51.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.045+0000 7f0b7965c700 1 -- 172.21.15.78:0/1343627009 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b740bff50 con 0x7f0b74154b90 2023-10-14T11:59:51.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.046+0000 7f0b5b7fe700 1 -- 172.21.15.78:0/1343627009 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0b6c004030 con 0x7f0b74154b90 2023-10-14T11:59:51.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.046+0000 7f0b5b7fe700 1 --2- 172.21.15.78:0/1343627009 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f0b6006af00 0x7f0b6006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:51.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.046+0000 7f0b5b7fe700 1 -- 172.21.15.78:0/1343627009 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f0b6c089660 con 0x7f0b74154b90 2023-10-14T11:59:51.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.046+0000 7f0b72ffd700 1 --2- 172.21.15.78:0/1343627009 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f0b6006af00 0x7f0b6006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:51.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.046+0000 7f0b72ffd700 1 --2- 172.21.15.78:0/1343627009 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f0b6006af00 0x7f0b6006d3c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0b680090b0 tx=0x7f0b68008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:51.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.050+0000 7f0b5b7fe700 1 -- 172.21.15.78:0/1343627009 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0b740bff50 con 0x7f0b74154b90 2023-10-14T11:59:51.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.240+0000 7f0b7965c700 1 -- 172.21.15.78:0/1343627009 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b740bff50 con 0x7f0b74154b90 2023-10-14T11:59:51.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.241+0000 7f0b5b7fe700 1 -- 172.21.15.78:0/1343627009 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+16565 (secure 0 0 0) 0x7f0b740bff50 con 0x7f0b74154b90 2023-10-14T11:59:51.243 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:59:51.244 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":32,"fsid":"4c5dc134-3092-406a-a1e1-13e6d9e926e8","created":"2023-10-14T11:57:57.363750+0000","modified":"2023-10-14T11:59:03.127845+0000","last_up_change":"2023-10-14T11:58:06.982472+0000","last_in_change":"2023-10-14T11:57:59.821373+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T11:58:09.001200+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"testcephfs1_metadata","create_time":"2023-10-14T11:58:59.469752+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"testcephfs1"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"testcephfs1_data","create_time":"2023-10-14T11:59:00.521599+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"testcephfs1"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"625a5fc0-9474-4eae-b80e-14c27a74ad19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6801","nonce":3884969780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6803","nonce":3884969780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6807","nonce":3884969780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6805","nonce":3884969780}]},"public_addr":"172.21.15.78:6801/3884969780","cluster_addr":"172.21.15.78:6803/3884969780","heartbeat_back_addr":"172.21.15.78:6807/3884969780","heartbeat_front_addr":"172.21.15.78:6805/3884969780","state":["exists","up"]},{"osd":1,"uuid":"49d63db1-2981-49ab-9c40-c50fc744b982","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6809","nonce":2176928406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6811","nonce":2176928406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6815","nonce":2176928406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6813","nonce":2176928406}]},"public_addr":"172.21.15.78:6809/2176928406","cluster_addr":"172.21.15.78:6811/2176928406","heartbeat_back_addr":"172.21.15.78:6815/2176928406","heartbeat_front_addr":"172.21.15.78:6813/2176928406","state":["exists","up"]},{"osd":2,"uuid":"1d7d0b29-cbd1-4717-9fe6-765a8d4998b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6825","nonce":3647044577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6827","nonce":3647044577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6831","nonce":3647044577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6829","nonce":3647044577}]},"public_addr":"172.21.15.78:6825/3647044577","cluster_addr":"172.21.15.78:6827/3647044577","heartbeat_back_addr":"172.21.15.78:6831/3647044577","heartbeat_front_addr":"172.21.15.78:6829/3647044577","state":["exists","up"]},{"osd":3,"uuid":"2f75ded1-e9e2-40e7-99ae-45e1e425e1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6817","nonce":3584112288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6819","nonce":3584112288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6823","nonce":3584112288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6821","nonce":3584112288}]},"public_addr":"172.21.15.78:6817/3584112288","cluster_addr":"172.21.15.78:6819/3584112288","heartbeat_back_addr":"172.21.15.78:6823/3584112288","heartbeat_front_addr":"172.21.15.78:6821/3584112288","state":["exists","up"]},{"osd":4,"uuid":"5130b432-8358-46cb-8a31-3064c976f6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6801","nonce":3732756301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6803","nonce":3732756301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6807","nonce":3732756301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6805","nonce":3732756301}]},"public_addr":"172.21.15.195:6801/3732756301","cluster_addr":"172.21.15.195:6803/3732756301","heartbeat_back_addr":"172.21.15.195:6807/3732756301","heartbeat_front_addr":"172.21.15.195:6805/3732756301","state":["exists","up"]},{"osd":5,"uuid":"60e234a9-a506-4e40-9c19-49a58f3cf28c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6825","nonce":2857706411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6827","nonce":2857706411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6831","nonce":2857706411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6829","nonce":2857706411}]},"public_addr":"172.21.15.195:6825/2857706411","cluster_addr":"172.21.15.195:6827/2857706411","heartbeat_back_addr":"172.21.15.195:6831/2857706411","heartbeat_front_addr":"172.21.15.195:6829/2857706411","state":["exists","up"]},{"osd":6,"uuid":"df7d2dd6-26b6-4eb1-96e5-94d1f3955b33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6810","nonce":72554374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6813","nonce":72554374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6820","nonce":72554374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6817","nonce":72554374}]},"public_addr":"172.21.15.195:6810/72554374","cluster_addr":"172.21.15.195:6813/72554374","heartbeat_back_addr":"172.21.15.195:6820/72554374","heartbeat_front_addr":"172.21.15.195:6817/72554374","state":["exists","up"]},{"osd":7,"uuid":"47a9d4de-f56d-4aa4-9900-c4a9cf9b2f97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6809","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6811","nonce":482472979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6816","nonce":482472979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6823","nonce":482472979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6821","nonce":482472979}]},"public_addr":"172.21.15.195:6811/482472979","cluster_addr":"172.21.15.195:6816/482472979","heartbeat_back_addr":"172.21.15.195:6823/482472979","heartbeat_front_addr":"172.21.15.195:6821/482472979","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.754003+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.005994+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.043603+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.024506+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.062585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.033767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.994825+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.038300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T11:59:51.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.243+0000 7f0b7965c700 1 -- 172.21.15.78:0/1343627009 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f0b6006af00 msgr2=0x7f0b6006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:51.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.243+0000 7f0b7965c700 1 --2- 172.21.15.78:0/1343627009 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f0b6006af00 0x7f0b6006d3c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0b680090b0 tx=0x7f0b68008030 comp rx=0 tx=0).stop 2023-10-14T11:59:51.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.243+0000 7f0b7965c700 1 -- 172.21.15.78:0/1343627009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b74154b90 msgr2=0x7f0b740c0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:51.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.243+0000 7f0b7965c700 1 --2- 172.21.15.78:0/1343627009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b74154b90 0x7f0b740c0770 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c007ef0 tx=0x7f0b6c00d3b0 comp rx=0 tx=0).stop 2023-10-14T11:59:51.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.243+0000 7f0b7965c700 1 -- 172.21.15.78:0/1343627009 shutdown_connections 2023-10-14T11:59:51.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.243+0000 7f0b7965c700 1 --2- 172.21.15.78:0/1343627009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b74154b90 0x7f0b740c0770 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:51.247 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.243+0000 7f0b7965c700 1 --2- 172.21.15.78:0/1343627009 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0b7406cd00 0x7f0b74147c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:51.247 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.243+0000 7f0b7965c700 1 --2- 172.21.15.78:0/1343627009 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f0b6006af00 0x7f0b6006d3c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:51.247 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.243+0000 7f0b7965c700 1 --2- 172.21.15.78:0/1343627009 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b7414b7e0 0x7f0b740c0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:51.247 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.243+0000 7f0b7965c700 1 -- 172.21.15.78:0/1343627009 >> 172.21.15.78:0/1343627009 conn(0x7f0b740ca2c0 msgr2=0x7f0b74152bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:51.247 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.243+0000 7f0b7965c700 1 -- 172.21.15.78:0/1343627009 shutdown_connections 2023-10-14T11:59:51.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.243+0000 7f0b7965c700 1 -- 172.21.15.78:0/1343627009 wait complete. 2023-10-14T11:59:51.257 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T11:58:09.001200+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'testcephfs1_metadata', 'create_time': '2023-10-14T11:58:59.469752+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '31', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'testcephfs1'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'testcephfs1_data', 'create_time': '2023-10-14T11:59:00.521599+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'testcephfs1'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T11:59:51.258 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T11:59:51.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.379+0000 7feb519b6700 1 Processor -- start 2023-10-14T11:59:51.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.379+0000 7feb519b6700 1 -- start start 2023-10-14T11:59:51.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.379+0000 7feb519b6700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7feb4c0c40f0 0x7feb4c1552d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:51.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.379+0000 7feb519b6700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7feb4c0a7ba0 0x7feb4c0c65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:51.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.379+0000 7feb519b6700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7feb4c151c00 0x7feb4c152070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:51.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.379+0000 7feb519b6700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb4c0c2400 con 0x7feb4c0a7ba0 2023-10-14T11:59:51.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.379+0000 7feb519b6700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb4c155810 con 0x7feb4c0c40f0 2023-10-14T11:59:51.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.379+0000 7feb519b6700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb4c0c0830 con 0x7feb4c151c00 2023-10-14T11:59:51.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.379+0000 7feb4a7fc700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7feb4c0a7ba0 0x7feb4c0c65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:51.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.379+0000 7feb4a7fc700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7feb4c0a7ba0 0x7feb4c0c65c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46902/0 (socket says 172.21.15.78:46902) 2023-10-14T11:59:51.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.379+0000 7feb4a7fc700 1 -- 172.21.15.78:0/49911691 learned_addr learned my addr 172.21.15.78:0/49911691 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:51.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.380+0000 7feb4b7fe700 1 --2- 172.21.15.78:0/49911691 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7feb4c151c00 0x7feb4c152070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:51.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.380+0000 7feb4affd700 1 --2- 172.21.15.78:0/49911691 >> v2:172.21.15.195:3300/0 conn(0x7feb4c0c40f0 0x7feb4c1552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:51.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.380+0000 7feb4a7fc700 1 -- 172.21.15.78:0/49911691 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7feb4c151c00 msgr2=0x7feb4c152070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:51.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.380+0000 7feb4a7fc700 1 --2- 172.21.15.78:0/49911691 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7feb4c151c00 0x7feb4c152070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:51.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.380+0000 7feb4a7fc700 1 -- 172.21.15.78:0/49911691 >> v2:172.21.15.195:3300/0 conn(0x7feb4c0c40f0 msgr2=0x7feb4c1552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:51.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.380+0000 7feb4a7fc700 1 --2- 172.21.15.78:0/49911691 >> v2:172.21.15.195:3300/0 conn(0x7feb4c0c40f0 0x7feb4c1552d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:51.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.380+0000 7feb4a7fc700 1 -- 172.21.15.78:0/49911691 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb4c0c4c60 con 0x7feb4c0a7ba0 2023-10-14T11:59:51.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.380+0000 7feb4a7fc700 1 --2- 172.21.15.78:0/49911691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb4c0a7ba0 0x7feb4c0c65c0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7feb4c0a8940 tx=0x7feb3c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=425f0affd21b3708 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:51.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.380+0000 7feb49ffb700 1 -- 172.21.15.78:0/49911691 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3c00b870 con 0x7feb4c0a7ba0 2023-10-14T11:59:51.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.380+0000 7feb49ffb700 1 -- 172.21.15.78:0/49911691 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb3c00b9d0 con 0x7feb4c0a7ba0 2023-10-14T11:59:51.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.380+0000 7feb49ffb700 1 -- 172.21.15.78:0/49911691 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3c00bd60 con 0x7feb4c0a7ba0 2023-10-14T11:59:51.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.381+0000 7feb519b6700 1 -- 172.21.15.78:0/49911691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb4c0a7ba0 msgr2=0x7feb4c0c65c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:51.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.381+0000 7feb519b6700 1 --2- 172.21.15.78:0/49911691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb4c0a7ba0 0x7feb4c0c65c0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7feb4c0a8940 tx=0x7feb3c009bd0 comp rx=0 tx=0).stop 2023-10-14T11:59:51.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.381+0000 7feb519b6700 1 -- 172.21.15.78:0/49911691 shutdown_connections 2023-10-14T11:59:51.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.381+0000 7feb519b6700 1 --2- 172.21.15.78:0/49911691 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7feb4c151c00 0x7feb4c152070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:51.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.381+0000 7feb519b6700 1 --2- 172.21.15.78:0/49911691 >> v2:172.21.15.195:3300/0 conn(0x7feb4c0c40f0 0x7feb4c1552d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:51.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.381+0000 7feb519b6700 1 --2- 172.21.15.78:0/49911691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb4c0a7ba0 0x7feb4c0c65c0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:51.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.381+0000 7feb519b6700 1 -- 172.21.15.78:0/49911691 >> 172.21.15.78:0/49911691 conn(0x7feb4c06af90 msgr2=0x7feb4c0c5eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:51.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.381+0000 7feb519b6700 1 -- 172.21.15.78:0/49911691 shutdown_connections 2023-10-14T11:59:51.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.381+0000 7feb519b6700 1 -- 172.21.15.78:0/49911691 wait complete. 2023-10-14T11:59:51.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.382+0000 7feb519b6700 1 Processor -- start 2023-10-14T11:59:51.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.382+0000 7feb519b6700 1 -- start start 2023-10-14T11:59:51.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.383+0000 7feb519b6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb4c0a7ba0 0x7feb4c0bc320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:51.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.383+0000 7feb519b6700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7feb4c0c40f0 0x7feb4c1ed210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:51.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.383+0000 7feb519b6700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb4c151c00 0x7feb4c14e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:51.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.383+0000 7feb519b6700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4c0b9930 con 0x7feb4c0a7ba0 2023-10-14T11:59:51.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.383+0000 7feb519b6700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4c0bf0a0 con 0x7feb4c151c00 2023-10-14T11:59:51.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.383+0000 7feb519b6700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb4c0bf380 con 0x7feb4c0c40f0 2023-10-14T11:59:51.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.383+0000 7feb4affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb4c0a7ba0 0x7feb4c0bc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:51.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.383+0000 7feb4affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb4c0a7ba0 0x7feb4c0bc320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46908/0 (socket says 172.21.15.78:46908) 2023-10-14T11:59:51.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.383+0000 7feb4affd700 1 -- 172.21.15.78:0/1296540294 learned_addr learned my addr 172.21.15.78:0/1296540294 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:51.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.383+0000 7feb4b7fe700 1 --2- 172.21.15.78:0/1296540294 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb4c151c00 0x7feb4c14e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:51.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.383+0000 7feb4a7fc700 1 --2- 172.21.15.78:0/1296540294 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7feb4c0c40f0 0x7feb4c1ed210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:51.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.383+0000 7feb4affd700 1 -- 172.21.15.78:0/1296540294 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7feb4c0c40f0 msgr2=0x7feb4c1ed210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:51.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.383+0000 7feb4affd700 1 --2- 172.21.15.78:0/1296540294 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7feb4c0c40f0 0x7feb4c1ed210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:51.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.383+0000 7feb4affd700 1 -- 172.21.15.78:0/1296540294 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb4c151c00 msgr2=0x7feb4c14e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:51.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.383+0000 7feb4affd700 1 --2- 172.21.15.78:0/1296540294 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb4c151c00 0x7feb4c14e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:51.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.383+0000 7feb4affd700 1 -- 172.21.15.78:0/1296540294 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb3c009550 con 0x7feb4c0a7ba0 2023-10-14T11:59:51.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.383+0000 7feb4affd700 1 --2- 172.21.15.78:0/1296540294 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb4c0a7ba0 0x7feb4c0bc320 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7feb3400e810 tx=0x7feb3400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:51.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.383+0000 7feb43fff700 1 -- 172.21.15.78:0/1296540294 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3400c070 con 0x7feb4c0a7ba0 2023-10-14T11:59:51.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.383+0000 7feb43fff700 1 -- 172.21.15.78:0/1296540294 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb34013610 con 0x7feb4c0a7ba0 2023-10-14T11:59:51.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.383+0000 7feb519b6700 1 -- 172.21.15.78:0/1296540294 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb4c0bf660 con 0x7feb4c0a7ba0 2023-10-14T11:59:51.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.383+0000 7feb43fff700 1 -- 172.21.15.78:0/1296540294 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb340139a0 con 0x7feb4c0a7ba0 2023-10-14T11:59:51.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.383+0000 7feb519b6700 1 -- 172.21.15.78:0/1296540294 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb4c1edc00 con 0x7feb4c0a7ba0 2023-10-14T11:59:51.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.384+0000 7feb519b6700 1 -- 172.21.15.78:0/1296540294 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb4c074b80 con 0x7feb4c0a7ba0 2023-10-14T11:59:51.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.385+0000 7feb43fff700 1 -- 172.21.15.78:0/1296540294 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7feb34013b00 con 0x7feb4c0a7ba0 2023-10-14T11:59:51.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.385+0000 7feb43fff700 1 --2- 172.21.15.78:0/1296540294 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7feb2c06af00 0x7feb2c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:51.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.385+0000 7feb43fff700 1 -- 172.21.15.78:0/1296540294 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7feb34089210 con 0x7feb4c0a7ba0 2023-10-14T11:59:51.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.385+0000 7feb4a7fc700 1 --2- 172.21.15.78:0/1296540294 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7feb2c06af00 0x7feb2c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:51.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.385+0000 7feb4a7fc700 1 --2- 172.21.15.78:0/1296540294 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7feb2c06af00 0x7feb2c06d3c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7feb3c014040 tx=0x7feb3c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:51.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.389+0000 7feb43fff700 1 -- 172.21.15.78:0/1296540294 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7feb34054330 con 0x7feb4c0a7ba0 2023-10-14T11:59:51.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.578+0000 7feb519b6700 1 -- 172.21.15.78:0/1296540294 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7feb4c0bf1e0 con 0x7feb4c0a7ba0 2023-10-14T11:59:51.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.579+0000 7feb43fff700 1 -- 172.21.15.78:0/1296540294 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v32) v1 ==== 93+0+10 (secure 0 0 0) 0x7feb4c0bf1e0 con 0x7feb4c0a7ba0 2023-10-14T11:59:51.581 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-10-14T11:59:51.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.581+0000 7feb519b6700 1 -- 172.21.15.78:0/1296540294 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7feb2c06af00 msgr2=0x7feb2c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:51.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.581+0000 7feb519b6700 1 --2- 172.21.15.78:0/1296540294 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7feb2c06af00 0x7feb2c06d3c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7feb3c014040 tx=0x7feb3c008030 comp rx=0 tx=0).stop 2023-10-14T11:59:51.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.581+0000 7feb519b6700 1 -- 172.21.15.78:0/1296540294 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb4c0a7ba0 msgr2=0x7feb4c0bc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:51.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.581+0000 7feb519b6700 1 --2- 172.21.15.78:0/1296540294 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb4c0a7ba0 0x7feb4c0bc320 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7feb3400e810 tx=0x7feb3400eb20 comp rx=0 tx=0).stop 2023-10-14T11:59:51.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.581+0000 7feb519b6700 1 -- 172.21.15.78:0/1296540294 shutdown_connections 2023-10-14T11:59:51.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.581+0000 7feb519b6700 1 --2- 172.21.15.78:0/1296540294 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7feb2c06af00 0x7feb2c06d3c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:51.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.581+0000 7feb519b6700 1 --2- 172.21.15.78:0/1296540294 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb4c151c00 0x7feb4c14e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:51.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.581+0000 7feb519b6700 1 --2- 172.21.15.78:0/1296540294 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb4c0a7ba0 0x7feb4c0bc320 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:51.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.581+0000 7feb519b6700 1 --2- 172.21.15.78:0/1296540294 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7feb4c0c40f0 0x7feb4c1ed210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:51.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.581+0000 7feb519b6700 1 -- 172.21.15.78:0/1296540294 >> 172.21.15.78:0/1296540294 conn(0x7feb4c06af90 msgr2=0x7feb4c14fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:51.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.581+0000 7feb519b6700 1 -- 172.21.15.78:0/1296540294 shutdown_connections 2023-10-14T11:59:51.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.581+0000 7feb519b6700 1 -- 172.21.15.78:0/1296540294 wait complete. 2023-10-14T11:59:51.595 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get testcephfs1_metadata pg_num 2023-10-14T11:59:51.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.716+0000 7f9f01f0d700 1 Processor -- start 2023-10-14T11:59:51.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.716+0000 7f9f01f0d700 1 -- start start 2023-10-14T11:59:51.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.717+0000 7f9f01f0d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9efc06f810 0x7f9efc156510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:51.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.717+0000 7f9f01f0d700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9efc06fca0 con 0x7f9efc06f430 2023-10-14T11:59:51.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.717+0000 7f9f01f0d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9efc156a50 con 0x7f9efc072390 2023-10-14T11:59:51.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.717+0000 7f9f01f0d700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9efc152850 con 0x7f9efc06f810 2023-10-14T11:59:51.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.717+0000 7f9efb7fe700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f9efc06f430 0x7f9efc13b810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51792/0 (socket says 172.21.15.78:51792) 2023-10-14T11:59:51.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.717+0000 7f9efb7fe700 1 -- 172.21.15.78:0/1172464442 learned_addr learned my addr 172.21.15.78:0/1172464442 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:59:51.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.717+0000 7f9efbfff700 1 --2- 172.21.15.78:0/1172464442 >> v2:172.21.15.195:3300/0 conn(0x7f9efc06f810 0x7f9efc156510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:51.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.717+0000 7f9efa7fc700 1 -- 172.21.15.78:0/1172464442 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3735334943 0 0) 0x7f9eec001110 con 0x7f9efc06f430 2023-10-14T11:59:51.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.717+0000 7f9efa7fc700 1 -- 172.21.15.78:0/1172464442 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ee40035b0 con 0x7f9efc06f430 2023-10-14T11:59:51.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.717+0000 7f9efbfff700 1 -- 172.21.15.78:0/1172464442 >> v1:172.21.15.195:6789/0 conn(0x7f9efc072390 legacy=0x7f9efc155e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:51.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.717+0000 7f9efbfff700 1 -- 172.21.15.78:0/1172464442 >> v1:172.21.15.78:6789/0 conn(0x7f9efc06f430 legacy=0x7f9efc13b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:51.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.717+0000 7f9efa7fc700 1 -- 172.21.15.78:0/1172464442 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 99394219 0 0) 0x7f9ee40035b0 con 0x7f9efc06f430 2023-10-14T11:59:51.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.717+0000 7f9efbfff700 1 -- 172.21.15.78:0/1172464442 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9efc13bf20 con 0x7f9efc06f810 2023-10-14T11:59:51.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.717+0000 7f9efbfff700 1 --2- 172.21.15.78:0/1172464442 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9efc06f810 0x7f9efc156510 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4009d90 tx=0x7f9ef400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afcbfd82b3b3d0bc server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:51.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.718+0000 7f9efa7fc700 1 -- 172.21.15.78:0/1172464442 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef400e070 con 0x7f9efc06f810 2023-10-14T11:59:51.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.718+0000 7f9efa7fc700 1 -- 172.21.15.78:0/1172464442 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ef400bb20 con 0x7f9efc06f810 2023-10-14T11:59:51.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.718+0000 7f9efa7fc700 1 -- 172.21.15.78:0/1172464442 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef400bed0 con 0x7f9efc06f810 2023-10-14T11:59:51.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.718+0000 7f9f01f0d700 1 -- 172.21.15.78:0/1172464442 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9efc06f810 msgr2=0x7f9efc156510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:51.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.718+0000 7f9f01f0d700 1 --2- 172.21.15.78:0/1172464442 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9efc06f810 0x7f9efc156510 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4009d90 tx=0x7f9ef400b330 comp rx=0 tx=0).stop 2023-10-14T11:59:51.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.718+0000 7f9f01f0d700 1 -- 172.21.15.78:0/1172464442 shutdown_connections 2023-10-14T11:59:51.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.718+0000 7f9f01f0d700 1 --2- 172.21.15.78:0/1172464442 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9efc06f810 0x7f9efc156510 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:51.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.718+0000 7f9f01f0d700 1 -- 172.21.15.78:0/1172464442 >> 172.21.15.78:0/1172464442 conn(0x7f9efc0c3a50 msgr2=0x7f9efc0c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:51.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.718+0000 7f9f01f0d700 1 -- 172.21.15.78:0/1172464442 shutdown_connections 2023-10-14T11:59:51.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.718+0000 7f9f01f0d700 1 -- 172.21.15.78:0/1172464442 wait complete. 2023-10-14T11:59:51.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.719+0000 7f9f01f0d700 1 Processor -- start 2023-10-14T11:59:51.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.719+0000 7f9f01f0d700 1 -- start start 2023-10-14T11:59:51.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.720+0000 7f9f01f0d700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9efc06f430 0x7f9efc151ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:51.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.720+0000 7f9f01f0d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9efc06f810 0x7f9efc1ea590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:51.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.720+0000 7f9f01f0d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9efc072390 0x7f9efc1eaad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:51.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.720+0000 7f9f01f0d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc06f270 con 0x7f9efc06f810 2023-10-14T11:59:51.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.720+0000 7f9f01f0d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc1504c0 con 0x7f9efc072390 2023-10-14T11:59:51.731 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.720+0000 7f9f01f0d700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc1ed4f0 con 0x7f9efc06f430 2023-10-14T11:59:51.731 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.720+0000 7f9efaffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9efc06f810 0x7f9efc1ea590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:51.731 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.720+0000 7f9efaffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9efc06f810 0x7f9efc1ea590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46922/0 (socket says 172.21.15.78:46922) 2023-10-14T11:59:51.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.720+0000 7f9efaffd700 1 -- 172.21.15.78:0/4184198380 learned_addr learned my addr 172.21.15.78:0/4184198380 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:51.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.720+0000 7f9efaffd700 1 -- 172.21.15.78:0/4184198380 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9efc06f430 msgr2=0x7f9efc151ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:51.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.720+0000 7f9efaffd700 1 --2- 172.21.15.78:0/4184198380 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9efc06f430 0x7f9efc151ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:51.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.720+0000 7f9efaffd700 1 -- 172.21.15.78:0/4184198380 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9efc072390 msgr2=0x7f9efc1eaad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:51.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.721+0000 7f9efaffd700 1 --2- 172.21.15.78:0/4184198380 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9efc072390 0x7f9efc1eaad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:51.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.721+0000 7f9efaffd700 1 -- 172.21.15.78:0/4184198380 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ef4009a70 con 0x7f9efc06f810 2023-10-14T11:59:51.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.721+0000 7f9efaffd700 1 --2- 172.21.15.78:0/4184198380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9efc06f810 0x7f9efc1ea590 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f9ef000c420 tx=0x7f9ef000c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:51.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.721+0000 7f9ee3fff700 1 -- 172.21.15.78:0/4184198380 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef00052d0 con 0x7f9efc06f810 2023-10-14T11:59:51.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.721+0000 7f9ee3fff700 1 -- 172.21.15.78:0/4184198380 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ef0005430 con 0x7f9efc06f810 2023-10-14T11:59:51.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.721+0000 7f9ee3fff700 1 -- 172.21.15.78:0/4184198380 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef00038c0 con 0x7f9efc06f810 2023-10-14T11:59:51.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.721+0000 7f9f01f0d700 1 -- 172.21.15.78:0/4184198380 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9efc0c0ef0 con 0x7f9efc06f810 2023-10-14T11:59:51.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.721+0000 7f9f01f0d700 1 -- 172.21.15.78:0/4184198380 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9efc151ff0 con 0x7f9efc06f810 2023-10-14T11:59:51.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.722+0000 7f9f01f0d700 1 -- 172.21.15.78:0/4184198380 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9efc1ed6c0 con 0x7f9efc06f810 2023-10-14T11:59:51.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.722+0000 7f9ee3fff700 1 -- 172.21.15.78:0/4184198380 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f9ef0003a20 con 0x7f9efc06f810 2023-10-14T11:59:51.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.723+0000 7f9ee3fff700 1 --2- 172.21.15.78:0/4184198380 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f9ee806af00 0x7f9ee806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:51.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.723+0000 7f9ee3fff700 1 -- 172.21.15.78:0/4184198380 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f9ef0088990 con 0x7f9efc06f810 2023-10-14T11:59:51.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.723+0000 7f9efb7fe700 1 --2- 172.21.15.78:0/4184198380 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f9ee806af00 0x7f9ee806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:51.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.723+0000 7f9efb7fe700 1 --2- 172.21.15.78:0/4184198380 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f9ee806af00 0x7f9ee806d3c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9eec005910 tx=0x7f9eec000fe0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:51.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.727+0000 7f9ee3fff700 1 -- 172.21.15.78:0/4184198380 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9efc1ed6c0 con 0x7f9efc06f810 2023-10-14T11:59:51.919 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.917+0000 7f9f01f0d700 1 -- 172.21.15.78:0/4184198380 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "testcephfs1_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9efc1ed6c0 con 0x7f9efc06f810 2023-10-14T11:59:51.919 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.917+0000 7f9ee3fff700 1 -- 172.21.15.78:0/4184198380 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testcephfs1_metadata", "var": "pg_num"}]=0 v32) v1 ==== 109+0+11 (secure 0 0 0) 0x7f9efc1ed6c0 con 0x7f9efc06f810 2023-10-14T11:59:51.920 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-10-14T11:59:51.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.919+0000 7f9f01f0d700 1 -- 172.21.15.78:0/4184198380 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f9ee806af00 msgr2=0x7f9ee806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:51.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.919+0000 7f9f01f0d700 1 --2- 172.21.15.78:0/4184198380 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f9ee806af00 0x7f9ee806d3c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9eec005910 tx=0x7f9eec000fe0 comp rx=0 tx=0).stop 2023-10-14T11:59:51.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.919+0000 7f9f01f0d700 1 -- 172.21.15.78:0/4184198380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9efc06f810 msgr2=0x7f9efc1ea590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:51.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.919+0000 7f9f01f0d700 1 --2- 172.21.15.78:0/4184198380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9efc06f810 0x7f9efc1ea590 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f9ef000c420 tx=0x7f9ef000c7e0 comp rx=0 tx=0).stop 2023-10-14T11:59:51.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.920+0000 7f9f01f0d700 1 -- 172.21.15.78:0/4184198380 shutdown_connections 2023-10-14T11:59:51.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.920+0000 7f9f01f0d700 1 --2- 172.21.15.78:0/4184198380 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f9ee806af00 0x7f9ee806d3c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:51.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.920+0000 7f9f01f0d700 1 --2- 172.21.15.78:0/4184198380 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9efc072390 0x7f9efc1eaad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:51.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.920+0000 7f9f01f0d700 1 --2- 172.21.15.78:0/4184198380 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9efc06f430 0x7f9efc151ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:51.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.920+0000 7f9f01f0d700 1 --2- 172.21.15.78:0/4184198380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9efc06f810 0x7f9efc1ea590 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:51.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.920+0000 7f9f01f0d700 1 -- 172.21.15.78:0/4184198380 >> 172.21.15.78:0/4184198380 conn(0x7f9efc0c3a50 msgr2=0x7f9efc0aa840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:51.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.920+0000 7f9f01f0d700 1 -- 172.21.15.78:0/4184198380 shutdown_connections 2023-10-14T11:59:51.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:51.920+0000 7f9f01f0d700 1 -- 172.21.15.78:0/4184198380 wait complete. 2023-10-14T11:59:51.933 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get testcephfs1_data pg_num 2023-10-14T11:59:52.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.055+0000 7f8b0f3ba700 1 Processor -- start 2023-10-14T11:59:52.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.055+0000 7f8b0f3ba700 1 -- start start 2023-10-14T11:59:52.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.056+0000 7f8b0f3ba700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8b0814ecc0 0x7f8b08151730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:52.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.056+0000 7f8b0f3ba700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b0806e080 con 0x7f8b0806da10 2023-10-14T11:59:52.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.056+0000 7f8b0f3ba700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b0806e360 con 0x7f8b080c1c00 2023-10-14T11:59:52.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.056+0000 7f8b0f3ba700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b0806de70 con 0x7f8b0814ecc0 2023-10-14T11:59:52.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.056+0000 7f8b0d156700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8b0814ecc0 0x7f8b08151730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:52.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.056+0000 7f8b0c955700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f8b0806da10 0x7f8b0806eef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51808/0 (socket says 172.21.15.78:51808) 2023-10-14T11:59:52.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.056+0000 7f8b0d156700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8b0814ecc0 0x7f8b08151730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46930/0 (socket says 172.21.15.78:46930) 2023-10-14T11:59:52.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.056+0000 7f8b0c955700 1 -- 172.21.15.78:0/1529921503 learned_addr learned my addr 172.21.15.78:0/1529921503 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:59:52.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.056+0000 7f8b0d156700 1 -- 172.21.15.78:0/1529921503 >> v1:172.21.15.195:6789/0 conn(0x7f8b080c1c00 legacy=0x7f8b080c2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:52.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.056+0000 7f8b0d156700 1 -- 172.21.15.78:0/1529921503 >> v1:172.21.15.78:6789/0 conn(0x7f8b0806da10 legacy=0x7f8b0806eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:52.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.056+0000 7f8b0d156700 1 -- 172.21.15.78:0/1529921503 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b0806e640 con 0x7f8b0814ecc0 2023-10-14T11:59:52.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.056+0000 7f8b0d156700 1 --2- 172.21.15.78:0/1529921503 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b0814ecc0 0x7f8b08151730 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f8b04009840 tx=0x7f8b04009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37547fbf3bcca7e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:52.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.056+0000 7f8afffff700 1 -- 172.21.15.78:0/1529921503 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b04011830 con 0x7f8b0814ecc0 2023-10-14T11:59:52.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.056+0000 7f8afffff700 1 -- 172.21.15.78:0/1529921503 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b04011990 con 0x7f8b0814ecc0 2023-10-14T11:59:52.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.056+0000 7f8afffff700 1 -- 172.21.15.78:0/1529921503 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b04011d20 con 0x7f8b0814ecc0 2023-10-14T11:59:52.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.057+0000 7f8b0f3ba700 1 -- 172.21.15.78:0/1529921503 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b0814ecc0 msgr2=0x7f8b08151730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:52.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.057+0000 7f8b0f3ba700 1 --2- 172.21.15.78:0/1529921503 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b0814ecc0 0x7f8b08151730 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f8b04009840 tx=0x7f8b04009b50 comp rx=0 tx=0).stop 2023-10-14T11:59:52.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.057+0000 7f8b0f3ba700 1 -- 172.21.15.78:0/1529921503 shutdown_connections 2023-10-14T11:59:52.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.057+0000 7f8b0f3ba700 1 --2- 172.21.15.78:0/1529921503 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b0814ecc0 0x7f8b08151730 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:52.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.057+0000 7f8b0f3ba700 1 -- 172.21.15.78:0/1529921503 >> 172.21.15.78:0/1529921503 conn(0x7f8b080c3a20 msgr2=0x7f8b080c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:52.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.057+0000 7f8b0f3ba700 1 -- 172.21.15.78:0/1529921503 shutdown_connections 2023-10-14T11:59:52.069 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.057+0000 7f8b0f3ba700 1 -- 172.21.15.78:0/1529921503 wait complete. 2023-10-14T11:59:52.069 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.058+0000 7f8b0f3ba700 1 Processor -- start 2023-10-14T11:59:52.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.058+0000 7f8b0f3ba700 1 -- start start 2023-10-14T11:59:52.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.059+0000 7f8b0f3ba700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8b0806da10 0x7f8b08151ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:52.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.059+0000 7f8b0f3ba700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8b080c1c00 0x7f8b081505f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:52.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.059+0000 7f8b0f3ba700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b081513a0 0x7f8b0814bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:52.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.059+0000 7f8b0f3ba700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b08150b30 con 0x7f8b081513a0 2023-10-14T11:59:52.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.059+0000 7f8b0f3ba700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b08151960 con 0x7f8b080c1c00 2023-10-14T11:59:52.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.059+0000 7f8b0f3ba700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b08144440 con 0x7f8b0806da10 2023-10-14T11:59:52.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.059+0000 7f8b0d957700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b081513a0 0x7f8b0814bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:52.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.059+0000 7f8b0d957700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b081513a0 0x7f8b0814bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46938/0 (socket says 172.21.15.78:46938) 2023-10-14T11:59:52.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.059+0000 7f8b0d957700 1 -- 172.21.15.78:0/873975595 learned_addr learned my addr 172.21.15.78:0/873975595 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:52.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.059+0000 7f8b0d156700 1 --2- 172.21.15.78:0/873975595 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8b0806da10 0x7f8b08151ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:52.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.059+0000 7f8b0c955700 1 --2- 172.21.15.78:0/873975595 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8b080c1c00 0x7f8b081505f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:52.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.059+0000 7f8b0d957700 1 -- 172.21.15.78:0/873975595 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8b0806da10 msgr2=0x7f8b08151ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:52.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.059+0000 7f8b0d957700 1 --2- 172.21.15.78:0/873975595 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8b0806da10 0x7f8b08151ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:52.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.059+0000 7f8b0d957700 1 -- 172.21.15.78:0/873975595 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8b080c1c00 msgr2=0x7f8b081505f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:52.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.059+0000 7f8b0d957700 1 --2- 172.21.15.78:0/873975595 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8b080c1c00 0x7f8b081505f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:52.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.059+0000 7f8b0d957700 1 -- 172.21.15.78:0/873975595 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b040094a0 con 0x7f8b081513a0 2023-10-14T11:59:52.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.059+0000 7f8b0c955700 1 --2- 172.21.15.78:0/873975595 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8b080c1c00 0x7f8b081505f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:59:52.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.059+0000 7f8b0d957700 1 --2- 172.21.15.78:0/873975595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b081513a0 0x7f8b0814bcd0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f8b0000cb10 tx=0x7f8b0000ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:52.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.059+0000 7f8afdffb700 1 -- 172.21.15.78:0/873975595 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b00003c50 con 0x7f8b081513a0 2023-10-14T11:59:52.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.059+0000 7f8afdffb700 1 -- 172.21.15.78:0/873975595 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b00003db0 con 0x7f8b081513a0 2023-10-14T11:59:52.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.059+0000 7f8b0f3ba700 1 -- 172.21.15.78:0/873975595 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b081ebe10 con 0x7f8b081513a0 2023-10-14T11:59:52.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.059+0000 7f8afdffb700 1 -- 172.21.15.78:0/873975595 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0000aa80 con 0x7f8b081513a0 2023-10-14T11:59:52.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.060+0000 7f8b0f3ba700 1 -- 172.21.15.78:0/873975595 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b08154fb0 con 0x7f8b081513a0 2023-10-14T11:59:52.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.061+0000 7f8b0f3ba700 1 -- 172.21.15.78:0/873975595 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b081ec050 con 0x7f8b081513a0 2023-10-14T11:59:52.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.061+0000 7f8afdffb700 1 -- 172.21.15.78:0/873975595 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8b0000abe0 con 0x7f8b081513a0 2023-10-14T11:59:52.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.061+0000 7f8afdffb700 1 --2- 172.21.15.78:0/873975595 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8aec06af00 0x7f8aec06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:52.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.061+0000 7f8afdffb700 1 -- 172.21.15.78:0/873975595 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f8b00089560 con 0x7f8b081513a0 2023-10-14T11:59:52.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.061+0000 7f8b0d156700 1 --2- 172.21.15.78:0/873975595 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8aec06af00 0x7f8aec06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:52.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.062+0000 7f8b0d156700 1 --2- 172.21.15.78:0/873975595 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8aec06af00 0x7f8aec06d3c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8b08072ac0 tx=0x7f8b04008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:52.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.066+0000 7f8afdffb700 1 -- 172.21.15.78:0/873975595 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8b081ec050 con 0x7f8b081513a0 2023-10-14T11:59:52.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.256+0000 7f8b0f3ba700 1 -- 172.21.15.78:0/873975595 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "testcephfs1_data", "var": "pg_num"} v 0) v1 -- 0x7f8b081ec050 con 0x7f8b081513a0 2023-10-14T11:59:52.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.256+0000 7f8afdffb700 1 -- 172.21.15.78:0/873975595 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testcephfs1_data", "var": "pg_num"}]=0 v32) v1 ==== 105+0+11 (secure 0 0 0) 0x7f8b081ec050 con 0x7f8b081513a0 2023-10-14T11:59:52.260 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2023-10-14T11:59:52.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.258+0000 7f8b0f3ba700 1 -- 172.21.15.78:0/873975595 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8aec06af00 msgr2=0x7f8aec06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:52.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.258+0000 7f8b0f3ba700 1 --2- 172.21.15.78:0/873975595 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8aec06af00 0x7f8aec06d3c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8b08072ac0 tx=0x7f8b04008030 comp rx=0 tx=0).stop 2023-10-14T11:59:52.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.258+0000 7f8b0f3ba700 1 -- 172.21.15.78:0/873975595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b081513a0 msgr2=0x7f8b0814bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:52.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.258+0000 7f8b0f3ba700 1 --2- 172.21.15.78:0/873975595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b081513a0 0x7f8b0814bcd0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f8b0000cb10 tx=0x7f8b0000ce20 comp rx=0 tx=0).stop 2023-10-14T11:59:52.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.259+0000 7f8b0f3ba700 1 -- 172.21.15.78:0/873975595 shutdown_connections 2023-10-14T11:59:52.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.259+0000 7f8b0f3ba700 1 --2- 172.21.15.78:0/873975595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b081513a0 0x7f8b0814bcd0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:52.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.259+0000 7f8b0f3ba700 1 --2- 172.21.15.78:0/873975595 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8b0806da10 0x7f8b08151ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:52.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.259+0000 7f8b0f3ba700 1 --2- 172.21.15.78:0/873975595 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8aec06af00 0x7f8aec06d3c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:52.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.259+0000 7f8b0f3ba700 1 --2- 172.21.15.78:0/873975595 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8b080c1c00 0x7f8b081505f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:52.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.259+0000 7f8b0f3ba700 1 -- 172.21.15.78:0/873975595 >> 172.21.15.78:0/873975595 conn(0x7f8b080c3a20 msgr2=0x7f8b080aa200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:52.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.259+0000 7f8b0f3ba700 1 -- 172.21.15.78:0/873975595 shutdown_connections 2023-10-14T11:59:52.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.259+0000 7f8b0f3ba700 1 -- 172.21.15.78:0/873975595 wait complete. 2023-10-14T11:59:52.272 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T11:59:52.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.391+0000 7f37a0a10700 1 Processor -- start 2023-10-14T11:59:52.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.391+0000 7f37a0a10700 1 -- start start 2023-10-14T11:59:52.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.392+0000 7f37a0a10700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f379c06f7b0 0x7f379c072360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:52.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.392+0000 7f37a0a10700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f379c0c1be0 0x7f379c14f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:52.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.392+0000 7f37a0a10700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f379c14fd80 0x7f379c1526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:52.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.392+0000 7f37a0a10700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f379c1501a0 con 0x7f379c0c1be0 2023-10-14T11:59:52.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.392+0000 7f37a0a10700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f379c155c90 con 0x7f379c14fd80 2023-10-14T11:59:52.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.392+0000 7f37a0a10700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f379c155f70 con 0x7f379c06f7b0 2023-10-14T11:59:52.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.392+0000 7f3799d9b700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f379c0c1be0 0x7f379c14f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:52.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.392+0000 7f3799d9b700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f379c0c1be0 0x7f379c14f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46946/0 (socket says 172.21.15.78:46946) 2023-10-14T11:59:52.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.392+0000 7f3799d9b700 1 -- 172.21.15.78:0/501686512 learned_addr learned my addr 172.21.15.78:0/501686512 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:52.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.392+0000 7f379ad9d700 1 --2- 172.21.15.78:0/501686512 >> v2:172.21.15.195:3300/0 conn(0x7f379c14fd80 0x7f379c1526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:52.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.392+0000 7f379a59c700 1 --2- 172.21.15.78:0/501686512 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f379c06f7b0 0x7f379c072360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:52.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.392+0000 7f3799d9b700 1 -- 172.21.15.78:0/501686512 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f379c06f7b0 msgr2=0x7f379c072360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:52.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.392+0000 7f3799d9b700 1 --2- 172.21.15.78:0/501686512 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f379c06f7b0 0x7f379c072360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:52.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.392+0000 7f3799d9b700 1 -- 172.21.15.78:0/501686512 >> v2:172.21.15.195:3300/0 conn(0x7f379c14fd80 msgr2=0x7f379c1526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:52.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.392+0000 7f3799d9b700 1 --2- 172.21.15.78:0/501686512 >> v2:172.21.15.195:3300/0 conn(0x7f379c14fd80 0x7f379c1526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:52.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.392+0000 7f3799d9b700 1 -- 172.21.15.78:0/501686512 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f379c156250 con 0x7f379c0c1be0 2023-10-14T11:59:52.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.392+0000 7f379ad9d700 1 --2- 172.21.15.78:0/501686512 >> v2:172.21.15.195:3300/0 conn(0x7f379c14fd80 0x7f379c1526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:59:52.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.393+0000 7f3799d9b700 1 --2- 172.21.15.78:0/501686512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f379c0c1be0 0x7f379c14f840 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f379c0c2060 tx=0x7f378c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75c06a7a001cf64b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:52.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.393+0000 7f379959a700 1 -- 172.21.15.78:0/501686512 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378c00b870 con 0x7f379c0c1be0 2023-10-14T11:59:52.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.393+0000 7f379959a700 1 -- 172.21.15.78:0/501686512 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f378c00b9d0 con 0x7f379c0c1be0 2023-10-14T11:59:52.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.393+0000 7f379959a700 1 -- 172.21.15.78:0/501686512 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378c00bd60 con 0x7f379c0c1be0 2023-10-14T11:59:52.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.393+0000 7f37a0a10700 1 -- 172.21.15.78:0/501686512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f379c0c1be0 msgr2=0x7f379c14f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:52.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.393+0000 7f37a0a10700 1 --2- 172.21.15.78:0/501686512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f379c0c1be0 0x7f379c14f840 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f379c0c2060 tx=0x7f378c009bd0 comp rx=0 tx=0).stop 2023-10-14T11:59:52.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.393+0000 7f37a0a10700 1 -- 172.21.15.78:0/501686512 shutdown_connections 2023-10-14T11:59:52.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.393+0000 7f37a0a10700 1 --2- 172.21.15.78:0/501686512 >> v2:172.21.15.195:3300/0 conn(0x7f379c14fd80 0x7f379c1526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:52.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.393+0000 7f37a0a10700 1 --2- 172.21.15.78:0/501686512 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f379c06f7b0 0x7f379c072360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:52.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.393+0000 7f37a0a10700 1 --2- 172.21.15.78:0/501686512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f379c0c1be0 0x7f379c14f840 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:52.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.393+0000 7f37a0a10700 1 -- 172.21.15.78:0/501686512 >> 172.21.15.78:0/501686512 conn(0x7f379c0c3ac0 msgr2=0x7f379c0c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:52.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.393+0000 7f37a0a10700 1 -- 172.21.15.78:0/501686512 shutdown_connections 2023-10-14T11:59:52.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.393+0000 7f37a0a10700 1 -- 172.21.15.78:0/501686512 wait complete. 2023-10-14T11:59:52.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.395+0000 7f37a0a10700 1 Processor -- start 2023-10-14T11:59:52.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.395+0000 7f37a0a10700 1 -- start start 2023-10-14T11:59:52.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.395+0000 7f37a0a10700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f379c06f7b0 0x7f379c1ebaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:52.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.395+0000 7f37a0a10700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f379c0c1be0 0x7f379c0aba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:52.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.395+0000 7f37a0a10700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f379c14fd80 0x7f379c145c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:52.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.395+0000 7f37a0a10700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379c14a5a0 con 0x7f379c14fd80 2023-10-14T11:59:52.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.395+0000 7f37a0a10700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379c152100 con 0x7f379c0c1be0 2023-10-14T11:59:52.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.395+0000 7f37a0a10700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f379c150740 con 0x7f379c06f7b0 2023-10-14T11:59:52.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.395+0000 7f379ad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f379c14fd80 0x7f379c145c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:52.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.395+0000 7f379ad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f379c14fd80 0x7f379c145c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46952/0 (socket says 172.21.15.78:46952) 2023-10-14T11:59:52.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.395+0000 7f379ad9d700 1 -- 172.21.15.78:0/1637747898 learned_addr learned my addr 172.21.15.78:0/1637747898 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:52.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.396+0000 7f379a59c700 1 --2- 172.21.15.78:0/1637747898 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f379c06f7b0 0x7f379c1ebaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:52.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.396+0000 7f3799d9b700 1 --2- 172.21.15.78:0/1637747898 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f379c0c1be0 0x7f379c0aba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:52.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.396+0000 7f379ad9d700 1 -- 172.21.15.78:0/1637747898 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f379c06f7b0 msgr2=0x7f379c1ebaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:52.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.396+0000 7f379ad9d700 1 --2- 172.21.15.78:0/1637747898 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f379c06f7b0 0x7f379c1ebaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:52.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.396+0000 7f379ad9d700 1 -- 172.21.15.78:0/1637747898 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f379c0c1be0 msgr2=0x7f379c0aba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:52.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.396+0000 7f379ad9d700 1 --2- 172.21.15.78:0/1637747898 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f379c0c1be0 0x7f379c0aba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:52.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.396+0000 7f379ad9d700 1 -- 172.21.15.78:0/1637747898 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f378c009550 con 0x7f379c14fd80 2023-10-14T11:59:52.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.396+0000 7f379ad9d700 1 --2- 172.21.15.78:0/1637747898 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f379c14fd80 0x7f379c145c10 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f3794007e70 tx=0x7f379400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:52.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.396+0000 7f378affd700 1 -- 172.21.15.78:0/1637747898 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379400dd20 con 0x7f379c14fd80 2023-10-14T11:59:52.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.396+0000 7f378affd700 1 -- 172.21.15.78:0/1637747898 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f379400f040 con 0x7f379c14fd80 2023-10-14T11:59:52.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.396+0000 7f37a0a10700 1 -- 172.21.15.78:0/1637747898 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f379c1ec030 con 0x7f379c14fd80 2023-10-14T11:59:52.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.396+0000 7f378affd700 1 -- 172.21.15.78:0/1637747898 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379400dea0 con 0x7f379c14fd80 2023-10-14T11:59:52.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.396+0000 7f37a0a10700 1 -- 172.21.15.78:0/1637747898 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f379c151ab0 con 0x7f379c14fd80 2023-10-14T11:59:52.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.397+0000 7f37a0a10700 1 -- 172.21.15.78:0/1637747898 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f379c151910 con 0x7f379c14fd80 2023-10-14T11:59:52.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.398+0000 7f378affd700 1 -- 172.21.15.78:0/1637747898 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3794003c60 con 0x7f379c14fd80 2023-10-14T11:59:52.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.398+0000 7f378affd700 1 --2- 172.21.15.78:0/1637747898 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f377c06af00 0x7f377c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:52.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.398+0000 7f378affd700 1 -- 172.21.15.78:0/1637747898 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f3794089780 con 0x7f379c14fd80 2023-10-14T11:59:52.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.398+0000 7f379a59c700 1 --2- 172.21.15.78:0/1637747898 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f377c06af00 0x7f377c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:52.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.399+0000 7f379a59c700 1 --2- 172.21.15.78:0/1637747898 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f377c06af00 0x7f377c06d3c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3790006b20 tx=0x7f3790008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:52.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.402+0000 7f378affd700 1 -- 172.21.15.78:0/1637747898 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f37940548f0 con 0x7f379c14fd80 2023-10-14T11:59:52.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.632+0000 7f37a0a10700 1 -- 172.21.15.78:0/1637747898 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f379c151910 con 0x7f379c14fd80 2023-10-14T11:59:52.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.632+0000 7f378affd700 1 -- 172.21.15.78:0/1637747898 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4924 (secure 0 0 0) 0x7f379c151910 con 0x7f379c14fd80 2023-10-14T11:59:52.635 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:59:52.635 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4658,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6834/1492486786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1492486786},{"type":"v1","addr":"172.21.15.195:6834","nonce":1492486786}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5016,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/2991384297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2991384297},{"type":"v1","addr":"172.21.15.78:6836","nonce":2991384297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5019,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3246640037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3246640037},{"type":"v1","addr":"172.21.15.78:6837","nonce":3246640037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T11:59:01.562764+0000","modified":"2023-10-14T11:59:04.592935+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5031":{"gid":5031,"name":"d","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.195:6835/2130671508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6833","nonce":2130671508},{"type":"v1","addr":"172.21.15.195:6835","nonce":2130671508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"testcephfs1","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-14T11:59:52.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.634+0000 7f37a0a10700 1 -- 172.21.15.78:0/1637747898 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f377c06af00 msgr2=0x7f377c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:52.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.634+0000 7f37a0a10700 1 --2- 172.21.15.78:0/1637747898 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f377c06af00 0x7f377c06d3c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3790006b20 tx=0x7f3790008040 comp rx=0 tx=0).stop 2023-10-14T11:59:52.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.634+0000 7f37a0a10700 1 -- 172.21.15.78:0/1637747898 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f379c14fd80 msgr2=0x7f379c145c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:52.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.634+0000 7f37a0a10700 1 --2- 172.21.15.78:0/1637747898 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f379c14fd80 0x7f379c145c10 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f3794007e70 tx=0x7f379400d3b0 comp rx=0 tx=0).stop 2023-10-14T11:59:52.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.634+0000 7f37a0a10700 1 -- 172.21.15.78:0/1637747898 shutdown_connections 2023-10-14T11:59:52.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.634+0000 7f37a0a10700 1 --2- 172.21.15.78:0/1637747898 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f379c14fd80 0x7f379c145c10 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:52.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.634+0000 7f37a0a10700 1 --2- 172.21.15.78:0/1637747898 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f379c06f7b0 0x7f379c1ebaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:52.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.634+0000 7f37a0a10700 1 --2- 172.21.15.78:0/1637747898 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f377c06af00 0x7f377c06d3c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:52.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.634+0000 7f37a0a10700 1 --2- 172.21.15.78:0/1637747898 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f379c0c1be0 0x7f379c0aba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:52.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.635+0000 7f37a0a10700 1 -- 172.21.15.78:0/1637747898 >> 172.21.15.78:0/1637747898 conn(0x7f379c0c3ac0 msgr2=0x7f379c0aa4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:52.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.635+0000 7f37a0a10700 1 -- 172.21.15.78:0/1637747898 shutdown_connections 2023-10-14T11:59:52.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.635+0000 7f37a0a10700 1 -- 172.21.15.78:0/1637747898 wait complete. 2023-10-14T11:59:52.639 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 17 2023-10-14T11:59:52.648 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T11:59:52.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.766+0000 7f0e48b44700 1 Processor -- start 2023-10-14T11:59:52.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.766+0000 7f0e48b44700 1 -- start start 2023-10-14T11:59:52.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.767+0000 7f0e48b44700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0e4406f1e0 0x7f0e441554e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:52.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.767+0000 7f0e48b44700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f0e440c0350 0x7f0e4413d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:52.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.767+0000 7f0e48b44700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e4406b630 con 0x7f0e4406ab40 2023-10-14T11:59:52.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.767+0000 7f0e48b44700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e440c49f0 con 0x7f0e4406f1e0 2023-10-14T11:59:52.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.767+0000 7f0e48b44700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e440726a0 con 0x7f0e440c0350 2023-10-14T11:59:52.772 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.767+0000 7f0e41d9b700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0e4406f1e0 0x7f0e441554e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:52.772 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.767+0000 7f0e4259c700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f0e4406ab40 0x7f0e4406af20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51832/0 (socket says 172.21.15.78:51832) 2023-10-14T11:59:52.772 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.767+0000 7f0e4259c700 1 -- 172.21.15.78:0/471611773 learned_addr learned my addr 172.21.15.78:0/471611773 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:59:52.772 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.767+0000 7f0e42d9d700 1 --2- 172.21.15.78:0/471611773 >> v2:172.21.15.195:3300/0 conn(0x7f0e440c0350 0x7f0e4413d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:52.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.768+0000 7f0e41d9b700 1 -- 172.21.15.78:0/471611773 >> v2:172.21.15.195:3300/0 conn(0x7f0e440c0350 msgr2=0x7f0e4413d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:52.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.768+0000 7f0e41d9b700 1 --2- 172.21.15.78:0/471611773 >> v2:172.21.15.195:3300/0 conn(0x7f0e440c0350 0x7f0e4413d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:52.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.768+0000 7f0e41d9b700 1 -- 172.21.15.78:0/471611773 >> v1:172.21.15.78:6789/0 conn(0x7f0e4406ab40 legacy=0x7f0e4406af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:52.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.768+0000 7f0e41d9b700 1 -- 172.21.15.78:0/471611773 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e44072980 con 0x7f0e4406f1e0 2023-10-14T11:59:52.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.768+0000 7f0e4159a700 1 -- 172.21.15.78:0/471611773 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 680097368 0 0) 0x7f0e4406b630 con 0x7f0e4406ab40 2023-10-14T11:59:52.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.768+0000 7f0e41d9b700 1 --2- 172.21.15.78:0/471611773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e4406f1e0 0x7f0e441554e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f0e440c1860 tx=0x7f0e34009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd05fd6a546c2314 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:52.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.768+0000 7f0e4159a700 1 -- 172.21.15.78:0/471611773 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e34011830 con 0x7f0e4406f1e0 2023-10-14T11:59:52.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.768+0000 7f0e4159a700 1 -- 172.21.15.78:0/471611773 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e34011990 con 0x7f0e4406f1e0 2023-10-14T11:59:52.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.768+0000 7f0e4159a700 1 -- 172.21.15.78:0/471611773 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e34011d20 con 0x7f0e4406f1e0 2023-10-14T11:59:52.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.768+0000 7f0e48b44700 1 -- 172.21.15.78:0/471611773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e4406f1e0 msgr2=0x7f0e441554e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:52.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.768+0000 7f0e48b44700 1 --2- 172.21.15.78:0/471611773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e4406f1e0 0x7f0e441554e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f0e440c1860 tx=0x7f0e34009b20 comp rx=0 tx=0).stop 2023-10-14T11:59:52.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.769+0000 7f0e48b44700 1 -- 172.21.15.78:0/471611773 shutdown_connections 2023-10-14T11:59:52.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.769+0000 7f0e48b44700 1 --2- 172.21.15.78:0/471611773 >> v2:172.21.15.195:3300/0 conn(0x7f0e440c0350 0x7f0e4413d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:52.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.769+0000 7f0e48b44700 1 --2- 172.21.15.78:0/471611773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e4406f1e0 0x7f0e441554e0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:52.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.769+0000 7f0e48b44700 1 -- 172.21.15.78:0/471611773 >> 172.21.15.78:0/471611773 conn(0x7f0e440c3ac0 msgr2=0x7f0e440c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:52.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.769+0000 7f0e48b44700 1 -- 172.21.15.78:0/471611773 shutdown_connections 2023-10-14T11:59:52.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.769+0000 7f0e48b44700 1 -- 172.21.15.78:0/471611773 wait complete. 2023-10-14T11:59:52.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.770+0000 7f0e48b44700 1 Processor -- start 2023-10-14T11:59:52.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.770+0000 7f0e48b44700 1 -- start start 2023-10-14T11:59:52.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.770+0000 7f0e48b44700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0e4406ab40 0x7f0e441ed830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:52.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.771+0000 7f0e48b44700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e4406f1e0 0x7f0e4413a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:52.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.771+0000 7f0e48b44700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0e440c0350 0x7f0e4414efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:52.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.771+0000 7f0e48b44700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e44151110 con 0x7f0e4406f1e0 2023-10-14T11:59:52.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.771+0000 7f0e48b44700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e441ed410 con 0x7f0e4406ab40 2023-10-14T11:59:52.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.771+0000 7f0e48b44700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e44132610 con 0x7f0e440c0350 2023-10-14T11:59:52.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.771+0000 7f0e41d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e4406f1e0 0x7f0e4413a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:52.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.771+0000 7f0e41d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e4406f1e0 0x7f0e4413a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46964/0 (socket says 172.21.15.78:46964) 2023-10-14T11:59:52.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.771+0000 7f0e41d9b700 1 -- 172.21.15.78:0/4054852804 learned_addr learned my addr 172.21.15.78:0/4054852804 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:52.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.771+0000 7f0e4259c700 1 --2- 172.21.15.78:0/4054852804 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0e4406ab40 0x7f0e441ed830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:52.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.771+0000 7f0e42d9d700 1 --2- 172.21.15.78:0/4054852804 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0e440c0350 0x7f0e4414efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:52.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.771+0000 7f0e41d9b700 1 -- 172.21.15.78:0/4054852804 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0e440c0350 msgr2=0x7f0e4414efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:52.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.771+0000 7f0e41d9b700 1 --2- 172.21.15.78:0/4054852804 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0e440c0350 0x7f0e4414efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:52.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.771+0000 7f0e41d9b700 1 -- 172.21.15.78:0/4054852804 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0e4406ab40 msgr2=0x7f0e441ed830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:52.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.771+0000 7f0e41d9b700 1 --2- 172.21.15.78:0/4054852804 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0e4406ab40 0x7f0e441ed830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:52.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.771+0000 7f0e41d9b700 1 -- 172.21.15.78:0/4054852804 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e340094a0 con 0x7f0e4406f1e0 2023-10-14T11:59:52.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.771+0000 7f0e4259c700 1 --2- 172.21.15.78:0/4054852804 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0e4406ab40 0x7f0e441ed830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:59:52.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.771+0000 7f0e41d9b700 1 --2- 172.21.15.78:0/4054852804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e4406f1e0 0x7f0e4413a3d0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f0e340090b0 tx=0x7f0e34004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:52.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.771+0000 7f0e32ffd700 1 -- 172.21.15.78:0/4054852804 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e340038a0 con 0x7f0e4406f1e0 2023-10-14T11:59:52.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.771+0000 7f0e32ffd700 1 -- 172.21.15.78:0/4054852804 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e3401f070 con 0x7f0e4406f1e0 2023-10-14T11:59:52.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.771+0000 7f0e48b44700 1 -- 172.21.15.78:0/4054852804 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e44153940 con 0x7f0e4406f1e0 2023-10-14T11:59:52.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.771+0000 7f0e32ffd700 1 -- 172.21.15.78:0/4054852804 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e34003c20 con 0x7f0e4406f1e0 2023-10-14T11:59:52.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.771+0000 7f0e48b44700 1 -- 172.21.15.78:0/4054852804 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e441ecd50 con 0x7f0e4406f1e0 2023-10-14T11:59:52.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.772+0000 7f0e48b44700 1 -- 172.21.15.78:0/4054852804 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e441ed550 con 0x7f0e4406f1e0 2023-10-14T11:59:52.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.773+0000 7f0e32ffd700 1 -- 172.21.15.78:0/4054852804 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0e3401a430 con 0x7f0e4406f1e0 2023-10-14T11:59:52.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.773+0000 7f0e32ffd700 1 --2- 172.21.15.78:0/4054852804 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f0e2406af00 0x7f0e2406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:52.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.773+0000 7f0e32ffd700 1 -- 172.21.15.78:0/4054852804 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f0e34021070 con 0x7f0e4406f1e0 2023-10-14T11:59:52.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.773+0000 7f0e4259c700 1 --2- 172.21.15.78:0/4054852804 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f0e2406af00 0x7f0e2406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:52.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.774+0000 7f0e4259c700 1 --2- 172.21.15.78:0/4054852804 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f0e2406af00 0x7f0e2406d3c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0e38000f60 tx=0x7f0e38009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:52.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.778+0000 7f0e32ffd700 1 -- 172.21.15.78:0/4054852804 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0e34056010 con 0x7f0e4406f1e0 2023-10-14T11:59:52.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.967+0000 7f0e48b44700 1 -- 172.21.15.78:0/4054852804 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e44074b80 con 0x7f0e4406f1e0 2023-10-14T11:59:52.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.968+0000 7f0e32ffd700 1 -- 172.21.15.78:0/4054852804 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+16565 (secure 0 0 0) 0x7f0e34059630 con 0x7f0e4406f1e0 2023-10-14T11:59:52.971 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:59:52.971 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":32,"fsid":"4c5dc134-3092-406a-a1e1-13e6d9e926e8","created":"2023-10-14T11:57:57.363750+0000","modified":"2023-10-14T11:59:03.127845+0000","last_up_change":"2023-10-14T11:58:06.982472+0000","last_in_change":"2023-10-14T11:57:59.821373+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T11:58:09.001200+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"testcephfs1_metadata","create_time":"2023-10-14T11:58:59.469752+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"testcephfs1"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"testcephfs1_data","create_time":"2023-10-14T11:59:00.521599+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"testcephfs1"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"625a5fc0-9474-4eae-b80e-14c27a74ad19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6801","nonce":3884969780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6803","nonce":3884969780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6807","nonce":3884969780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6805","nonce":3884969780}]},"public_addr":"172.21.15.78:6801/3884969780","cluster_addr":"172.21.15.78:6803/3884969780","heartbeat_back_addr":"172.21.15.78:6807/3884969780","heartbeat_front_addr":"172.21.15.78:6805/3884969780","state":["exists","up"]},{"osd":1,"uuid":"49d63db1-2981-49ab-9c40-c50fc744b982","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6809","nonce":2176928406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6811","nonce":2176928406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6815","nonce":2176928406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6813","nonce":2176928406}]},"public_addr":"172.21.15.78:6809/2176928406","cluster_addr":"172.21.15.78:6811/2176928406","heartbeat_back_addr":"172.21.15.78:6815/2176928406","heartbeat_front_addr":"172.21.15.78:6813/2176928406","state":["exists","up"]},{"osd":2,"uuid":"1d7d0b29-cbd1-4717-9fe6-765a8d4998b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6825","nonce":3647044577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6827","nonce":3647044577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6831","nonce":3647044577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6829","nonce":3647044577}]},"public_addr":"172.21.15.78:6825/3647044577","cluster_addr":"172.21.15.78:6827/3647044577","heartbeat_back_addr":"172.21.15.78:6831/3647044577","heartbeat_front_addr":"172.21.15.78:6829/3647044577","state":["exists","up"]},{"osd":3,"uuid":"2f75ded1-e9e2-40e7-99ae-45e1e425e1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6817","nonce":3584112288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6819","nonce":3584112288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6823","nonce":3584112288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6821","nonce":3584112288}]},"public_addr":"172.21.15.78:6817/3584112288","cluster_addr":"172.21.15.78:6819/3584112288","heartbeat_back_addr":"172.21.15.78:6823/3584112288","heartbeat_front_addr":"172.21.15.78:6821/3584112288","state":["exists","up"]},{"osd":4,"uuid":"5130b432-8358-46cb-8a31-3064c976f6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6801","nonce":3732756301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6803","nonce":3732756301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6807","nonce":3732756301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6805","nonce":3732756301}]},"public_addr":"172.21.15.195:6801/3732756301","cluster_addr":"172.21.15.195:6803/3732756301","heartbeat_back_addr":"172.21.15.195:6807/3732756301","heartbeat_front_addr":"172.21.15.195:6805/3732756301","state":["exists","up"]},{"osd":5,"uuid":"60e234a9-a506-4e40-9c19-49a58f3cf28c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6825","nonce":2857706411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6827","nonce":2857706411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6831","nonce":2857706411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6829","nonce":2857706411}]},"public_addr":"172.21.15.195:6825/2857706411","cluster_addr":"172.21.15.195:6827/2857706411","heartbeat_back_addr":"172.21.15.195:6831/2857706411","heartbeat_front_addr":"172.21.15.195:6829/2857706411","state":["exists","up"]},{"osd":6,"uuid":"df7d2dd6-26b6-4eb1-96e5-94d1f3955b33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6810","nonce":72554374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6813","nonce":72554374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6820","nonce":72554374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6817","nonce":72554374}]},"public_addr":"172.21.15.195:6810/72554374","cluster_addr":"172.21.15.195:6813/72554374","heartbeat_back_addr":"172.21.15.195:6820/72554374","heartbeat_front_addr":"172.21.15.195:6817/72554374","state":["exists","up"]},{"osd":7,"uuid":"47a9d4de-f56d-4aa4-9900-c4a9cf9b2f97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6809","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6811","nonce":482472979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6816","nonce":482472979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6823","nonce":482472979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6821","nonce":482472979}]},"public_addr":"172.21.15.195:6811/482472979","cluster_addr":"172.21.15.195:6816/482472979","heartbeat_back_addr":"172.21.15.195:6823/482472979","heartbeat_front_addr":"172.21.15.195:6821/482472979","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.754003+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.005994+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.043603+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.024506+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.062585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.033767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.994825+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.038300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T11:59:52.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.971+0000 7f0e48b44700 1 -- 172.21.15.78:0/4054852804 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f0e2406af00 msgr2=0x7f0e2406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:52.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.971+0000 7f0e48b44700 1 --2- 172.21.15.78:0/4054852804 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f0e2406af00 0x7f0e2406d3c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0e38000f60 tx=0x7f0e38009040 comp rx=0 tx=0).stop 2023-10-14T11:59:52.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.971+0000 7f0e48b44700 1 -- 172.21.15.78:0/4054852804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e4406f1e0 msgr2=0x7f0e4413a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:52.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.971+0000 7f0e48b44700 1 --2- 172.21.15.78:0/4054852804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e4406f1e0 0x7f0e4413a3d0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f0e340090b0 tx=0x7f0e34004750 comp rx=0 tx=0).stop 2023-10-14T11:59:52.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.971+0000 7f0e48b44700 1 -- 172.21.15.78:0/4054852804 shutdown_connections 2023-10-14T11:59:52.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.971+0000 7f0e48b44700 1 --2- 172.21.15.78:0/4054852804 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0e440c0350 0x7f0e4414efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:52.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.971+0000 7f0e48b44700 1 --2- 172.21.15.78:0/4054852804 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f0e2406af00 0x7f0e2406d3c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:52.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.971+0000 7f0e48b44700 1 --2- 172.21.15.78:0/4054852804 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0e4406ab40 0x7f0e441ed830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:52.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.971+0000 7f0e48b44700 1 --2- 172.21.15.78:0/4054852804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e4406f1e0 0x7f0e4413a3d0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:52.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.971+0000 7f0e48b44700 1 -- 172.21.15.78:0/4054852804 >> 172.21.15.78:0/4054852804 conn(0x7f0e440c3ac0 msgr2=0x7f0e440c2150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:52.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.971+0000 7f0e48b44700 1 -- 172.21.15.78:0/4054852804 shutdown_connections 2023-10-14T11:59:52.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:52.971+0000 7f0e48b44700 1 -- 172.21.15.78:0/4054852804 wait complete. 2023-10-14T11:59:52.984 INFO:tasks.cephfs.filesystem:Destroying file system testcephfs1 and related pools 2023-10-14T11:59:52.985 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T11:59:53.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.103+0000 7f63897c0700 1 Processor -- start 2023-10-14T11:59:53.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.103+0000 7f63897c0700 1 -- start start 2023-10-14T11:59:53.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.104+0000 7f63897c0700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f63840c1be0 0x7f638414f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:53.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.104+0000 7f63897c0700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f638414fd80 0x7f63841526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:53.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.104+0000 7f63897c0700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6384155c90 con 0x7f638406f7b0 2023-10-14T11:59:53.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.104+0000 7f63897c0700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63841501a0 con 0x7f638414fd80 2023-10-14T11:59:53.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.104+0000 7f63897c0700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6384155f70 con 0x7f63840c1be0 2023-10-14T11:59:53.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.104+0000 7f63837fe700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f638414fd80 0x7f63841526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:53.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.104+0000 7f63837fe700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f638414fd80 0x7f63841526a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46974/0 (socket says 172.21.15.78:46974) 2023-10-14T11:59:53.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.104+0000 7f63837fe700 1 -- 172.21.15.78:0/3983234048 learned_addr learned my addr 172.21.15.78:0/3983234048 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:53.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.105+0000 7f63827fc700 1 --2- 172.21.15.78:0/3983234048 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f63840c1be0 0x7f638414f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:53.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.105+0000 7f63837fe700 1 -- 172.21.15.78:0/3983234048 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f63840c1be0 msgr2=0x7f638414f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:53.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.105+0000 7f63837fe700 1 --2- 172.21.15.78:0/3983234048 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f63840c1be0 0x7f638414f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:53.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.105+0000 7f63837fe700 1 -- 172.21.15.78:0/3983234048 >> v1:172.21.15.195:6789/0 conn(0x7f638406f7b0 legacy=0x7f6384072360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:53.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.105+0000 7f63837fe700 1 -- 172.21.15.78:0/3983234048 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6384156250 con 0x7f638414fd80 2023-10-14T11:59:53.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.105+0000 7f63827fc700 1 --2- 172.21.15.78:0/3983234048 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f63840c1be0 0x7f638414f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:59:53.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.105+0000 7f63837fe700 1 --2- 172.21.15.78:0/3983234048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f638414fd80 0x7f63841526a0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f637c009e50 tx=0x7f637c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dc31d79927d6a74 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:53.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.105+0000 7f6381ffb700 1 -- 172.21.15.78:0/3983234048 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637c00e070 con 0x7f638414fd80 2023-10-14T11:59:53.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.105+0000 7f6381ffb700 1 -- 172.21.15.78:0/3983234048 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f637c003a20 con 0x7f638414fd80 2023-10-14T11:59:53.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.105+0000 7f6381ffb700 1 -- 172.21.15.78:0/3983234048 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637c003dd0 con 0x7f638414fd80 2023-10-14T11:59:53.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.105+0000 7f63897c0700 1 -- 172.21.15.78:0/3983234048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f638414fd80 msgr2=0x7f63841526a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:53.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.105+0000 7f63897c0700 1 --2- 172.21.15.78:0/3983234048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f638414fd80 0x7f63841526a0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f637c009e50 tx=0x7f637c00b330 comp rx=0 tx=0).stop 2023-10-14T11:59:53.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.105+0000 7f63897c0700 1 -- 172.21.15.78:0/3983234048 shutdown_connections 2023-10-14T11:59:53.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.105+0000 7f63897c0700 1 --2- 172.21.15.78:0/3983234048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f638414fd80 0x7f63841526a0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:53.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.105+0000 7f63897c0700 1 --2- 172.21.15.78:0/3983234048 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f63840c1be0 0x7f638414f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:53.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.106+0000 7f63897c0700 1 -- 172.21.15.78:0/3983234048 >> 172.21.15.78:0/3983234048 conn(0x7f63840c3ac0 msgr2=0x7f63840c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:53.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.106+0000 7f63897c0700 1 -- 172.21.15.78:0/3983234048 shutdown_connections 2023-10-14T11:59:53.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.106+0000 7f63897c0700 1 -- 172.21.15.78:0/3983234048 wait complete. 2023-10-14T11:59:53.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.107+0000 7f63897c0700 1 Processor -- start 2023-10-14T11:59:53.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.107+0000 7f63897c0700 1 -- start start 2023-10-14T11:59:53.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.107+0000 7f63897c0700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f638406f7b0 0x7f6384146bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:53.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.107+0000 7f63897c0700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63840c1be0 0x7f638413f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:53.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.107+0000 7f63897c0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f638414fd80 0x7f638413fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:53.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.107+0000 7f63897c0700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6384147500 con 0x7f638414fd80 2023-10-14T11:59:53.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.107+0000 7f63897c0700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638414cae0 con 0x7f63840c1be0 2023-10-14T11:59:53.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.107+0000 7f63897c0700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f638414c800 con 0x7f638406f7b0 2023-10-14T11:59:53.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.108+0000 7f63837fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f638414fd80 0x7f638413fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:53.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.108+0000 7f63837fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f638414fd80 0x7f638413fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46980/0 (socket says 172.21.15.78:46980) 2023-10-14T11:59:53.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.108+0000 7f6382ffd700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f638406f7b0 0x7f6384146bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:53.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.108+0000 7f63837fe700 1 -- 172.21.15.78:0/273978740 learned_addr learned my addr 172.21.15.78:0/273978740 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:53.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.108+0000 7f63827fc700 1 --2- 172.21.15.78:0/273978740 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63840c1be0 0x7f638413f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:53.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.108+0000 7f63837fe700 1 -- 172.21.15.78:0/273978740 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f638406f7b0 msgr2=0x7f6384146bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:53.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.108+0000 7f63837fe700 1 --2- 172.21.15.78:0/273978740 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f638406f7b0 0x7f6384146bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:53.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.108+0000 7f63837fe700 1 -- 172.21.15.78:0/273978740 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63840c1be0 msgr2=0x7f638413f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:53.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.108+0000 7f63837fe700 1 --2- 172.21.15.78:0/273978740 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63840c1be0 0x7f638413f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:53.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.108+0000 7f63837fe700 1 -- 172.21.15.78:0/273978740 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f637c009b30 con 0x7f638414fd80 2023-10-14T11:59:53.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.108+0000 7f63837fe700 1 --2- 172.21.15.78:0/273978740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f638414fd80 0x7f638413fc80 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f637c015040 tx=0x7f637c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:53.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.108+0000 7f636b7fe700 1 -- 172.21.15.78:0/273978740 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637c004440 con 0x7f638414fd80 2023-10-14T11:59:53.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.108+0000 7f636b7fe700 1 -- 172.21.15.78:0/273978740 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f637c0045c0 con 0x7f638414fd80 2023-10-14T11:59:53.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.108+0000 7f63897c0700 1 -- 172.21.15.78:0/273978740 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6384149160 con 0x7f638414fd80 2023-10-14T11:59:53.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.108+0000 7f636b7fe700 1 -- 172.21.15.78:0/273978740 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637c004970 con 0x7f638414fd80 2023-10-14T11:59:53.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.108+0000 7f63897c0700 1 -- 172.21.15.78:0/273978740 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6384147100 con 0x7f638414fd80 2023-10-14T11:59:53.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.109+0000 7f63897c0700 1 -- 172.21.15.78:0/273978740 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6384147640 con 0x7f638414fd80 2023-10-14T11:59:53.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.110+0000 7f636b7fe700 1 -- 172.21.15.78:0/273978740 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f637c012430 con 0x7f638414fd80 2023-10-14T11:59:53.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.110+0000 7f636b7fe700 1 --2- 172.21.15.78:0/273978740 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f637006af00 0x7f637006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:53.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.110+0000 7f636b7fe700 1 -- 172.21.15.78:0/273978740 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f637c08a940 con 0x7f638414fd80 2023-10-14T11:59:53.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.110+0000 7f6382ffd700 1 --2- 172.21.15.78:0/273978740 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f637006af00 0x7f637006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:53.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.110+0000 7f6382ffd700 1 --2- 172.21.15.78:0/273978740 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f637006af00 0x7f637006d3c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6378007fd0 tx=0x7f6378007ec0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:53.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.115+0000 7f636b7fe700 1 -- 172.21.15.78:0/273978740 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6384147640 con 0x7f638414fd80 2023-10-14T11:59:53.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.347+0000 7f63897c0700 1 -- 172.21.15.78:0/273978740 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6384147640 con 0x7f638414fd80 2023-10-14T11:59:53.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.348+0000 7f636b7fe700 1 -- 172.21.15.78:0/273978740 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4924 (secure 0 0 0) 0x7f6384147640 con 0x7f638414fd80 2023-10-14T11:59:53.350 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:59:53.350 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4658,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6834/1492486786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1492486786},{"type":"v1","addr":"172.21.15.195:6834","nonce":1492486786}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5016,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/2991384297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2991384297},{"type":"v1","addr":"172.21.15.78:6836","nonce":2991384297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5019,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3246640037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3246640037},{"type":"v1","addr":"172.21.15.78:6837","nonce":3246640037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T11:59:01.562764+0000","modified":"2023-10-14T11:59:04.592935+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5031":{"gid":5031,"name":"d","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.195:6835/2130671508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6833","nonce":2130671508},{"type":"v1","addr":"172.21.15.195:6835","nonce":2130671508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"testcephfs1","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-14T11:59:53.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.351+0000 7f63897c0700 1 -- 172.21.15.78:0/273978740 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f637006af00 msgr2=0x7f637006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:53.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.351+0000 7f63897c0700 1 --2- 172.21.15.78:0/273978740 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f637006af00 0x7f637006d3c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6378007fd0 tx=0x7f6378007ec0 comp rx=0 tx=0).stop 2023-10-14T11:59:53.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.351+0000 7f63897c0700 1 -- 172.21.15.78:0/273978740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f638414fd80 msgr2=0x7f638413fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:53.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.351+0000 7f63897c0700 1 --2- 172.21.15.78:0/273978740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f638414fd80 0x7f638413fc80 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f637c015040 tx=0x7f637c004ad0 comp rx=0 tx=0).stop 2023-10-14T11:59:53.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.351+0000 7f63897c0700 1 -- 172.21.15.78:0/273978740 shutdown_connections 2023-10-14T11:59:53.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.351+0000 7f63897c0700 1 --2- 172.21.15.78:0/273978740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f638414fd80 0x7f638413fc80 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:53.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.351+0000 7f63897c0700 1 --2- 172.21.15.78:0/273978740 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f638406f7b0 0x7f6384146bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:53.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.351+0000 7f63897c0700 1 --2- 172.21.15.78:0/273978740 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f637006af00 0x7f637006d3c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:53.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.351+0000 7f63897c0700 1 --2- 172.21.15.78:0/273978740 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63840c1be0 0x7f638413f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:53.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.351+0000 7f63897c0700 1 -- 172.21.15.78:0/273978740 >> 172.21.15.78:0/273978740 conn(0x7f63840c3ac0 msgr2=0x7f63840ab6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:53.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.351+0000 7f63897c0700 1 -- 172.21.15.78:0/273978740 shutdown_connections 2023-10-14T11:59:53.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.351+0000 7f63897c0700 1 -- 172.21.15.78:0/273978740 wait complete. 2023-10-14T11:59:53.355 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 17 2023-10-14T11:59:53.364 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T11:59:53.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.482+0000 7f293adfe700 1 Processor -- start 2023-10-14T11:59:53.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.482+0000 7f293adfe700 1 -- start start 2023-10-14T11:59:53.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.483+0000 7f293adfe700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f293406ab40 0x7f293406f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:53.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.483+0000 7f293adfe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f29340c0350 0x7f293413d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:53.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.483+0000 7f293adfe700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f293406b630 con 0x7f29341554e0 2023-10-14T11:59:53.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.483+0000 7f293adfe700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29340c49f0 con 0x7f29340c0350 2023-10-14T11:59:53.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.483+0000 7f293adfe700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29340726a0 con 0x7f293406ab40 2023-10-14T11:59:53.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.483+0000 7f2933fff700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f29341554e0 0x7f293406af20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51860/0 (socket says 172.21.15.78:51860) 2023-10-14T11:59:53.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.483+0000 7f2933fff700 1 -- 172.21.15.78:0/2393906648 learned_addr learned my addr 172.21.15.78:0/2393906648 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:59:53.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.483+0000 7f29337fe700 1 -- 172.21.15.78:0/2393906648 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1354659139 0 0) 0x7f293406b630 con 0x7f29341554e0 2023-10-14T11:59:53.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.483+0000 7f29337fe700 1 -- 172.21.15.78:0/2393906648 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29240035b0 con 0x7f29341554e0 2023-10-14T11:59:53.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.483+0000 7f293939b700 1 --2- 172.21.15.78:0/2393906648 >> v2:172.21.15.195:3300/0 conn(0x7f29340c0350 0x7f293413d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:53.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.483+0000 7f2938b9a700 1 --2- 172.21.15.78:0/2393906648 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f293406ab40 0x7f293406f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:53.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.484+0000 7f29337fe700 1 -- 172.21.15.78:0/2393906648 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 133851696 0 0) 0x7f29240035b0 con 0x7f29341554e0 2023-10-14T11:59:53.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.484+0000 7f29337fe700 1 -- 172.21.15.78:0/2393906648 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f293406ab40 msgr2=0x7f293406f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:53.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.484+0000 7f29337fe700 1 --2- 172.21.15.78:0/2393906648 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f293406ab40 0x7f293406f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:53.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.484+0000 7f29337fe700 1 -- 172.21.15.78:0/2393906648 >> v2:172.21.15.195:3300/0 conn(0x7f29340c0350 msgr2=0x7f293413d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:53.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.484+0000 7f29337fe700 1 --2- 172.21.15.78:0/2393906648 >> v2:172.21.15.195:3300/0 conn(0x7f29340c0350 0x7f293413d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:53.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.484+0000 7f29337fe700 1 -- 172.21.15.78:0/2393906648 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2934072980 con 0x7f29341554e0 2023-10-14T11:59:53.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.484+0000 7f29337fe700 1 -- 172.21.15.78:0/2393906648 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441799917 0 0) 0x7f2920001670 con 0x7f29341554e0 2023-10-14T11:59:53.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.484+0000 7f2938b9a700 1 --2- 172.21.15.78:0/2393906648 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f293406ab40 0x7f293406f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:59:53.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.484+0000 7f29337fe700 1 -- 172.21.15.78:0/2393906648 >> v1:172.21.15.78:6789/0 conn(0x7f29341554e0 legacy=0x7f293406af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:53.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.484+0000 7f29337fe700 1 --2- 172.21.15.78:0/2393906648 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f293406ab40 0x7f2924003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:53.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.484+0000 7f29337fe700 1 --2- 172.21.15.78:0/2393906648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2924003f40 0x7f2924004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:53.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.484+0000 7f29337fe700 1 --2- 172.21.15.78:0/2393906648 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2924004d90 0x7f2924005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:53.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.484+0000 7f29337fe700 1 -- 172.21.15.78:0/2393906648 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29340c49f0 con 0x7f2924003f40 2023-10-14T11:59:53.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.484+0000 7f29337fe700 1 -- 172.21.15.78:0/2393906648 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29340726a0 con 0x7f2924004d90 2023-10-14T11:59:53.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.484+0000 7f29337fe700 1 -- 172.21.15.78:0/2393906648 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f292400bac0 con 0x7f293406ab40 2023-10-14T11:59:53.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.484+0000 7f2933fff700 1 --2- 172.21.15.78:0/2393906648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2924003f40 0x7f2924004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:53.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.484+0000 7f2938b9a700 1 --2- 172.21.15.78:0/2393906648 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f293406ab40 0x7f2924003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:53.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.484+0000 7f293939b700 1 --2- 172.21.15.78:0/2393906648 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2924004d90 0x7f2924005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:53.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.484+0000 7f2933fff700 1 -- 172.21.15.78:0/2393906648 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f293406ab40 msgr2=0x7f2924003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:53.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.484+0000 7f2933fff700 1 --2- 172.21.15.78:0/2393906648 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f293406ab40 0x7f2924003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:53.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.484+0000 7f2933fff700 1 -- 172.21.15.78:0/2393906648 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2924004d90 msgr2=0x7f2924005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:53.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.484+0000 7f2933fff700 1 --2- 172.21.15.78:0/2393906648 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2924004d90 0x7f2924005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:53.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.484+0000 7f2933fff700 1 -- 172.21.15.78:0/2393906648 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29240035b0 con 0x7f2924003f40 2023-10-14T11:59:53.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.484+0000 7f2933fff700 1 --2- 172.21.15.78:0/2393906648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2924003f40 0x7f2924004850 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f292000c320 tx=0x7f2924005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7954833151920582 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:53.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.484+0000 7f29337fe700 1 -- 172.21.15.78:0/2393906648 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292000ce60 con 0x7f2924003f40 2023-10-14T11:59:53.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.484+0000 7f29337fe700 1 -- 172.21.15.78:0/2393906648 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2920005960 con 0x7f2924003f40 2023-10-14T11:59:53.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.484+0000 7f29337fe700 1 -- 172.21.15.78:0/2393906648 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2920005d10 con 0x7f2924003f40 2023-10-14T11:59:53.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.485+0000 7f293adfe700 1 -- 172.21.15.78:0/2393906648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2924003f40 msgr2=0x7f2924004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:53.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.485+0000 7f293adfe700 1 --2- 172.21.15.78:0/2393906648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2924003f40 0x7f2924004850 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f292000c320 tx=0x7f2924005880 comp rx=0 tx=0).stop 2023-10-14T11:59:53.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.485+0000 7f293939b700 1 -- 172.21.15.78:0/2393906648 reap_dead start 2023-10-14T11:59:53.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.485+0000 7f293adfe700 1 -- 172.21.15.78:0/2393906648 shutdown_connections 2023-10-14T11:59:53.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.485+0000 7f293adfe700 1 -- 172.21.15.78:0/2393906648 >> 172.21.15.78:0/2393906648 conn(0x7f29340c3ac0 msgr2=0x7f29340c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:53.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.485+0000 7f293adfe700 1 -- 172.21.15.78:0/2393906648 shutdown_connections 2023-10-14T11:59:53.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.485+0000 7f293adfe700 1 -- 172.21.15.78:0/2393906648 wait complete. 2023-10-14T11:59:53.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.487+0000 7f293adfe700 1 Processor -- start 2023-10-14T11:59:53.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.487+0000 7f293adfe700 1 -- start start 2023-10-14T11:59:53.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.489+0000 7f293adfe700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2924003f40 0x7f293414efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:53.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.489+0000 7f293adfe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29341e9ac0 0x7f2934150630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:53.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.489+0000 7f293adfe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2934134d50 0x7f2934135ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:53.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.489+0000 7f293adfe700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2934137ef0 con 0x7f29341e9ac0 2023-10-14T11:59:53.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.489+0000 7f293adfe700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2934135200 con 0x7f2934134d50 2023-10-14T11:59:53.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.489+0000 7f293adfe700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29341501a0 con 0x7f2924003f40 2023-10-14T11:59:53.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.489+0000 7f2933fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29341e9ac0 0x7f2934150630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:53.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.489+0000 7f2933fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29341e9ac0 0x7f2934150630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46998/0 (socket says 172.21.15.78:46998) 2023-10-14T11:59:53.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.489+0000 7f2933fff700 1 -- 172.21.15.78:0/3196309270 learned_addr learned my addr 172.21.15.78:0/3196309270 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:53.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.489+0000 7f2938b9a700 1 --2- 172.21.15.78:0/3196309270 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2924003f40 0x7f293414efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:53.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.489+0000 7f293939b700 1 --2- 172.21.15.78:0/3196309270 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2934134d50 0x7f2934135ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:53.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.489+0000 7f2933fff700 1 -- 172.21.15.78:0/3196309270 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2924003f40 msgr2=0x7f293414efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:53.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.489+0000 7f2933fff700 1 --2- 172.21.15.78:0/3196309270 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2924003f40 0x7f293414efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:53.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.489+0000 7f2933fff700 1 -- 172.21.15.78:0/3196309270 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2934134d50 msgr2=0x7f2934135ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:53.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.489+0000 7f2933fff700 1 --2- 172.21.15.78:0/3196309270 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2934134d50 0x7f2934135ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:53.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.489+0000 7f2933fff700 1 -- 172.21.15.78:0/3196309270 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f293414f590 con 0x7f29341e9ac0 2023-10-14T11:59:53.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.489+0000 7f2933fff700 1 --2- 172.21.15.78:0/3196309270 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29341e9ac0 0x7f2934150630 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f292400ba10 tx=0x7f2920001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:53.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.489+0000 7f29317fa700 1 -- 172.21.15.78:0/3196309270 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2920001660 con 0x7f29341e9ac0 2023-10-14T11:59:53.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.489+0000 7f29317fa700 1 -- 172.21.15.78:0/3196309270 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2920004a80 con 0x7f29341e9ac0 2023-10-14T11:59:53.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.489+0000 7f293adfe700 1 -- 172.21.15.78:0/3196309270 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f293414f870 con 0x7f29341e9ac0 2023-10-14T11:59:53.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.489+0000 7f29317fa700 1 -- 172.21.15.78:0/3196309270 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2920004e30 con 0x7f29341e9ac0 2023-10-14T11:59:53.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.489+0000 7f293adfe700 1 -- 172.21.15.78:0/3196309270 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f293414fe30 con 0x7f29341e9ac0 2023-10-14T11:59:53.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.489+0000 7f293adfe700 1 -- 172.21.15.78:0/3196309270 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2934074b80 con 0x7f29341e9ac0 2023-10-14T11:59:53.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.490+0000 7f29317fa700 1 -- 172.21.15.78:0/3196309270 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f29200070d0 con 0x7f29341e9ac0 2023-10-14T11:59:53.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.490+0000 7f29317fa700 1 --2- 172.21.15.78:0/3196309270 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f291806af00 0x7f291806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:53.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.490+0000 7f29317fa700 1 -- 172.21.15.78:0/3196309270 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f292008abf0 con 0x7f29341e9ac0 2023-10-14T11:59:53.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.490+0000 7f2938b9a700 1 --2- 172.21.15.78:0/3196309270 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f291806af00 0x7f291806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:53.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.490+0000 7f2938b9a700 1 --2- 172.21.15.78:0/3196309270 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f291806af00 0x7f291806d3c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f29341ee170 tx=0x7f2928008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:53.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.495+0000 7f29317fa700 1 -- 172.21.15.78:0/3196309270 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2920055d10 con 0x7f29341e9ac0 2023-10-14T11:59:53.683 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T11:59:53.685 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T11:59:53.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.731+0000 7f293adfe700 1 -- 172.21.15.78:0/3196309270 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29341502e0 con 0x7f29341e9ac0 2023-10-14T11:59:53.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.731+0000 7f29317fa700 1 -- 172.21.15.78:0/3196309270 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4924 (secure 0 0 0) 0x7f2920007da0 con 0x7f29341e9ac0 2023-10-14T11:59:53.734 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:59:53.734 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4658,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6834/1492486786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1492486786},{"type":"v1","addr":"172.21.15.195:6834","nonce":1492486786}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5016,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/2991384297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2991384297},{"type":"v1","addr":"172.21.15.78:6836","nonce":2991384297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5019,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3246640037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3246640037},{"type":"v1","addr":"172.21.15.78:6837","nonce":3246640037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T11:59:01.562764+0000","modified":"2023-10-14T11:59:04.592935+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5031":{"gid":5031,"name":"d","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.195:6835/2130671508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6833","nonce":2130671508},{"type":"v1","addr":"172.21.15.195:6835","nonce":2130671508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"testcephfs1","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-14T11:59:53.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.734+0000 7f293adfe700 1 -- 172.21.15.78:0/3196309270 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f291806af00 msgr2=0x7f291806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:53.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.734+0000 7f293adfe700 1 --2- 172.21.15.78:0/3196309270 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f291806af00 0x7f291806d3c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f29341ee170 tx=0x7f2928008040 comp rx=0 tx=0).stop 2023-10-14T11:59:53.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.734+0000 7f293adfe700 1 -- 172.21.15.78:0/3196309270 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29341e9ac0 msgr2=0x7f2934150630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:53.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.734+0000 7f293adfe700 1 --2- 172.21.15.78:0/3196309270 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29341e9ac0 0x7f2934150630 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f292400ba10 tx=0x7f2920001a00 comp rx=0 tx=0).stop 2023-10-14T11:59:53.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.734+0000 7f293adfe700 1 -- 172.21.15.78:0/3196309270 shutdown_connections 2023-10-14T11:59:53.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.734+0000 7f293adfe700 1 --2- 172.21.15.78:0/3196309270 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f291806af00 0x7f291806d3c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:53.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.734+0000 7f293adfe700 1 --2- 172.21.15.78:0/3196309270 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2934134d50 0x7f2934135ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:53.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.734+0000 7f293adfe700 1 --2- 172.21.15.78:0/3196309270 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2924003f40 0x7f293414efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:53.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.734+0000 7f293adfe700 1 --2- 172.21.15.78:0/3196309270 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29341e9ac0 0x7f2934150630 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:53.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.734+0000 7f293adfe700 1 -- 172.21.15.78:0/3196309270 >> 172.21.15.78:0/3196309270 conn(0x7f29340c3ac0 msgr2=0x7f293406af20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:53.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.734+0000 7f293adfe700 1 -- 172.21.15.78:0/3196309270 shutdown_connections 2023-10-14T11:59:53.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.734+0000 7f293adfe700 1 -- 172.21.15.78:0/3196309270 wait complete. 2023-10-14T11:59:53.738 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 17 2023-10-14T11:59:53.750 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T11:59:53.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.909+0000 7fc3d7460700 1 Processor -- start 2023-10-14T11:59:53.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.909+0000 7fc3d7460700 1 -- start start 2023-10-14T11:59:53.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.909+0000 7fc3d7460700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fc3d014b5b0 0x7fc3d00c8240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:53.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.909+0000 7fc3d7460700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc3d006b420 0x7fc3d00c9b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:53.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.909+0000 7fc3d7460700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3d00ca060 con 0x7fc3d0155580 2023-10-14T11:59:53.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.909+0000 7fc3d7460700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3d00c8780 con 0x7fc3d006b420 2023-10-14T11:59:53.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.909+0000 7fc3d7460700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3d00c8970 con 0x7fc3d014b5b0 2023-10-14T11:59:53.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.910+0000 7fc3d49fb700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc3d006b420 0x7fc3d00c9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:53.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.910+0000 7fc3d49fb700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc3d006b420 0x7fc3d00c9b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47006/0 (socket says 172.21.15.78:47006) 2023-10-14T11:59:53.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.910+0000 7fc3d49fb700 1 -- 172.21.15.78:0/3343943090 learned_addr learned my addr 172.21.15.78:0/3343943090 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:53.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.910+0000 7fc3d49fb700 1 -- 172.21.15.78:0/3343943090 >> v2:172.21.15.195:3300/0 conn(0x7fc3d014b5b0 msgr2=0x7fc3d00c8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:53.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.910+0000 7fc3d49fb700 1 --2- 172.21.15.78:0/3343943090 >> v2:172.21.15.195:3300/0 conn(0x7fc3d014b5b0 0x7fc3d00c8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:53.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.910+0000 7fc3d49fb700 1 -- 172.21.15.78:0/3343943090 >> v1:172.21.15.195:6789/0 conn(0x7fc3d0155580 legacy=0x7fc3d00c4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:53.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.910+0000 7fc3d49fb700 1 -- 172.21.15.78:0/3343943090 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d00c9040 con 0x7fc3d006b420 2023-10-14T11:59:53.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.910+0000 7fc3d49fb700 1 --2- 172.21.15.78:0/3343943090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3d006b420 0x7fc3d00c9b20 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc3d00c7ec0 tx=0x7fc3cc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b791dbe4b3b2057 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:53.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.910+0000 7fc3c7fff700 1 -- 172.21.15.78:0/3343943090 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3cc004230 con 0x7fc3d006b420 2023-10-14T11:59:53.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.910+0000 7fc3c7fff700 1 -- 172.21.15.78:0/3343943090 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3cc004390 con 0x7fc3d006b420 2023-10-14T11:59:53.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.910+0000 7fc3c7fff700 1 -- 172.21.15.78:0/3343943090 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3cc0036a0 con 0x7fc3d006b420 2023-10-14T11:59:53.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.911+0000 7fc3d7460700 1 -- 172.21.15.78:0/3343943090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3d006b420 msgr2=0x7fc3d00c9b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:53.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.911+0000 7fc3d7460700 1 --2- 172.21.15.78:0/3343943090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3d006b420 0x7fc3d00c9b20 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc3d00c7ec0 tx=0x7fc3cc009bd0 comp rx=0 tx=0).stop 2023-10-14T11:59:53.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.911+0000 7fc3d7460700 1 -- 172.21.15.78:0/3343943090 shutdown_connections 2023-10-14T11:59:53.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.911+0000 7fc3d7460700 1 --2- 172.21.15.78:0/3343943090 >> v2:172.21.15.195:3300/0 conn(0x7fc3d014b5b0 0x7fc3d00c8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:53.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.911+0000 7fc3d7460700 1 --2- 172.21.15.78:0/3343943090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3d006b420 0x7fc3d00c9b20 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:53.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.911+0000 7fc3d7460700 1 -- 172.21.15.78:0/3343943090 >> 172.21.15.78:0/3343943090 conn(0x7fc3d00abe30 msgr2=0x7fc3d00ac240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:53.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.911+0000 7fc3d7460700 1 -- 172.21.15.78:0/3343943090 shutdown_connections 2023-10-14T11:59:53.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.911+0000 7fc3d7460700 1 -- 172.21.15.78:0/3343943090 wait complete. 2023-10-14T11:59:53.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.912+0000 7fc3d7460700 1 Processor -- start 2023-10-14T11:59:53.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.912+0000 7fc3d7460700 1 -- start start 2023-10-14T11:59:53.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.913+0000 7fc3d7460700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3d006b420 0x7fc3d0154650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:53.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.913+0000 7fc3d7460700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3d014b5b0 0x7fc3d01e9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:53.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.913+0000 7fc3d7460700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc3d0155580 0x7fc3d01e9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:53.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.913+0000 7fc3d7460700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d006c370 con 0x7fc3d006b420 2023-10-14T11:59:53.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.913+0000 7fc3d7460700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d01ea410 con 0x7fc3d014b5b0 2023-10-14T11:59:53.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.913+0000 7fc3d7460700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d01dcb30 con 0x7fc3d0155580 2023-10-14T11:59:53.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.913+0000 7fc3d51fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3d006b420 0x7fc3d0154650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:53.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.913+0000 7fc3d51fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3d006b420 0x7fc3d0154650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47010/0 (socket says 172.21.15.78:47010) 2023-10-14T11:59:53.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.913+0000 7fc3d51fc700 1 -- 172.21.15.78:0/3981523007 learned_addr learned my addr 172.21.15.78:0/3981523007 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:53.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.913+0000 7fc3d49fb700 1 --2- 172.21.15.78:0/3981523007 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3d014b5b0 0x7fc3d01e9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:53.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.913+0000 7fc3d59fd700 1 --2- 172.21.15.78:0/3981523007 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc3d0155580 0x7fc3d01e9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:53.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.913+0000 7fc3d51fc700 1 -- 172.21.15.78:0/3981523007 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc3d0155580 msgr2=0x7fc3d01e9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:53.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.913+0000 7fc3d51fc700 1 --2- 172.21.15.78:0/3981523007 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc3d0155580 0x7fc3d01e9ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:53.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.913+0000 7fc3d51fc700 1 -- 172.21.15.78:0/3981523007 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3d014b5b0 msgr2=0x7fc3d01e9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:53.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.913+0000 7fc3d51fc700 1 --2- 172.21.15.78:0/3981523007 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3d014b5b0 0x7fc3d01e9990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:53.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.913+0000 7fc3d51fc700 1 -- 172.21.15.78:0/3981523007 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3cc009550 con 0x7fc3d006b420 2023-10-14T11:59:53.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.913+0000 7fc3d59fd700 1 --2- 172.21.15.78:0/3981523007 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc3d0155580 0x7fc3d01e9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:59:53.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.913+0000 7fc3d51fc700 1 --2- 172.21.15.78:0/3981523007 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3d006b420 0x7fc3d0154650 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc3c000b530 tx=0x7fc3c000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:53.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.913+0000 7fc3c5ffb700 1 -- 172.21.15.78:0/3981523007 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c0011830 con 0x7fc3d006b420 2023-10-14T11:59:53.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.913+0000 7fc3c5ffb700 1 -- 172.21.15.78:0/3981523007 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3c00119b0 con 0x7fc3d006b420 2023-10-14T11:59:53.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.913+0000 7fc3d7460700 1 -- 172.21.15.78:0/3981523007 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3d01ecf00 con 0x7fc3d006b420 2023-10-14T11:59:53.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.913+0000 7fc3c5ffb700 1 -- 172.21.15.78:0/3981523007 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c0011d60 con 0x7fc3d006b420 2023-10-14T11:59:53.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.913+0000 7fc3d7460700 1 -- 172.21.15.78:0/3981523007 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d01ed4c0 con 0x7fc3d006b420 2023-10-14T11:59:53.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.914+0000 7fc3d7460700 1 -- 172.21.15.78:0/3981523007 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3d0074b80 con 0x7fc3d006b420 2023-10-14T11:59:53.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.915+0000 7fc3c5ffb700 1 -- 172.21.15.78:0/3981523007 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc3c001a400 con 0x7fc3d006b420 2023-10-14T11:59:53.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.915+0000 7fc3c5ffb700 1 --2- 172.21.15.78:0/3981523007 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc3b406af00 0x7fc3b406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:53.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.915+0000 7fc3c5ffb700 1 -- 172.21.15.78:0/3981523007 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7fc3c0089680 con 0x7fc3d006b420 2023-10-14T11:59:53.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.915+0000 7fc3d49fb700 1 --2- 172.21.15.78:0/3981523007 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc3b406af00 0x7fc3b406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:53.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.916+0000 7fc3d49fb700 1 --2- 172.21.15.78:0/3981523007 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc3b406af00 0x7fc3b406d3c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc009130 tx=0x7fc3cc008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:53.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:53.921+0000 7fc3c5ffb700 1 -- 172.21.15.78:0/3981523007 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc3c00547a0 con 0x7fc3d006b420 2023-10-14T11:59:54.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.110+0000 7fc3d7460700 1 -- 172.21.15.78:0/3981523007 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3d0155ba0 con 0x7fc3d006b420 2023-10-14T11:59:54.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.111+0000 7fc3c5ffb700 1 -- 172.21.15.78:0/3981523007 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+16565 (secure 0 0 0) 0x7fc3c0057dc0 con 0x7fc3d006b420 2023-10-14T11:59:54.113 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T11:59:54.114 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":32,"fsid":"4c5dc134-3092-406a-a1e1-13e6d9e926e8","created":"2023-10-14T11:57:57.363750+0000","modified":"2023-10-14T11:59:03.127845+0000","last_up_change":"2023-10-14T11:58:06.982472+0000","last_in_change":"2023-10-14T11:57:59.821373+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T11:58:09.001200+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"testcephfs1_metadata","create_time":"2023-10-14T11:58:59.469752+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"testcephfs1"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"testcephfs1_data","create_time":"2023-10-14T11:59:00.521599+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"testcephfs1"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"625a5fc0-9474-4eae-b80e-14c27a74ad19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6801","nonce":3884969780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6803","nonce":3884969780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6807","nonce":3884969780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6805","nonce":3884969780}]},"public_addr":"172.21.15.78:6801/3884969780","cluster_addr":"172.21.15.78:6803/3884969780","heartbeat_back_addr":"172.21.15.78:6807/3884969780","heartbeat_front_addr":"172.21.15.78:6805/3884969780","state":["exists","up"]},{"osd":1,"uuid":"49d63db1-2981-49ab-9c40-c50fc744b982","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6809","nonce":2176928406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6811","nonce":2176928406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6815","nonce":2176928406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6813","nonce":2176928406}]},"public_addr":"172.21.15.78:6809/2176928406","cluster_addr":"172.21.15.78:6811/2176928406","heartbeat_back_addr":"172.21.15.78:6815/2176928406","heartbeat_front_addr":"172.21.15.78:6813/2176928406","state":["exists","up"]},{"osd":2,"uuid":"1d7d0b29-cbd1-4717-9fe6-765a8d4998b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6825","nonce":3647044577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6827","nonce":3647044577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6831","nonce":3647044577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6829","nonce":3647044577}]},"public_addr":"172.21.15.78:6825/3647044577","cluster_addr":"172.21.15.78:6827/3647044577","heartbeat_back_addr":"172.21.15.78:6831/3647044577","heartbeat_front_addr":"172.21.15.78:6829/3647044577","state":["exists","up"]},{"osd":3,"uuid":"2f75ded1-e9e2-40e7-99ae-45e1e425e1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6817","nonce":3584112288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6819","nonce":3584112288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6823","nonce":3584112288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6821","nonce":3584112288}]},"public_addr":"172.21.15.78:6817/3584112288","cluster_addr":"172.21.15.78:6819/3584112288","heartbeat_back_addr":"172.21.15.78:6823/3584112288","heartbeat_front_addr":"172.21.15.78:6821/3584112288","state":["exists","up"]},{"osd":4,"uuid":"5130b432-8358-46cb-8a31-3064c976f6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6801","nonce":3732756301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6803","nonce":3732756301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6807","nonce":3732756301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6805","nonce":3732756301}]},"public_addr":"172.21.15.195:6801/3732756301","cluster_addr":"172.21.15.195:6803/3732756301","heartbeat_back_addr":"172.21.15.195:6807/3732756301","heartbeat_front_addr":"172.21.15.195:6805/3732756301","state":["exists","up"]},{"osd":5,"uuid":"60e234a9-a506-4e40-9c19-49a58f3cf28c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6825","nonce":2857706411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6827","nonce":2857706411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6831","nonce":2857706411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6829","nonce":2857706411}]},"public_addr":"172.21.15.195:6825/2857706411","cluster_addr":"172.21.15.195:6827/2857706411","heartbeat_back_addr":"172.21.15.195:6831/2857706411","heartbeat_front_addr":"172.21.15.195:6829/2857706411","state":["exists","up"]},{"osd":6,"uuid":"df7d2dd6-26b6-4eb1-96e5-94d1f3955b33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6810","nonce":72554374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6813","nonce":72554374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6820","nonce":72554374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6817","nonce":72554374}]},"public_addr":"172.21.15.195:6810/72554374","cluster_addr":"172.21.15.195:6813/72554374","heartbeat_back_addr":"172.21.15.195:6820/72554374","heartbeat_front_addr":"172.21.15.195:6817/72554374","state":["exists","up"]},{"osd":7,"uuid":"47a9d4de-f56d-4aa4-9900-c4a9cf9b2f97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6809","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6811","nonce":482472979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6816","nonce":482472979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6823","nonce":482472979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6821","nonce":482472979}]},"public_addr":"172.21.15.195:6811/482472979","cluster_addr":"172.21.15.195:6816/482472979","heartbeat_back_addr":"172.21.15.195:6823/482472979","heartbeat_front_addr":"172.21.15.195:6821/482472979","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.754003+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.005994+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.043603+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.024506+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.062585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.033767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.994825+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.038300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T11:59:54.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.114+0000 7fc3d7460700 1 -- 172.21.15.78:0/3981523007 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc3b406af00 msgr2=0x7fc3b406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:54.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.114+0000 7fc3d7460700 1 --2- 172.21.15.78:0/3981523007 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc3b406af00 0x7fc3b406d3c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc009130 tx=0x7fc3cc008030 comp rx=0 tx=0).stop 2023-10-14T11:59:54.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.114+0000 7fc3d7460700 1 -- 172.21.15.78:0/3981523007 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3d006b420 msgr2=0x7fc3d0154650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:54.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.114+0000 7fc3d7460700 1 --2- 172.21.15.78:0/3981523007 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3d006b420 0x7fc3d0154650 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc3c000b530 tx=0x7fc3c000b8f0 comp rx=0 tx=0).stop 2023-10-14T11:59:54.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.114+0000 7fc3d7460700 1 -- 172.21.15.78:0/3981523007 shutdown_connections 2023-10-14T11:59:54.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.114+0000 7fc3d7460700 1 --2- 172.21.15.78:0/3981523007 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc3d0155580 0x7fc3d01e9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:54.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.114+0000 7fc3d7460700 1 --2- 172.21.15.78:0/3981523007 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3d006b420 0x7fc3d0154650 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:54.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.114+0000 7fc3d7460700 1 --2- 172.21.15.78:0/3981523007 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fc3b406af00 0x7fc3b406d3c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:54.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.114+0000 7fc3d7460700 1 --2- 172.21.15.78:0/3981523007 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3d014b5b0 0x7fc3d01e9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:54.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.114+0000 7fc3d7460700 1 -- 172.21.15.78:0/3981523007 >> 172.21.15.78:0/3981523007 conn(0x7fc3d00abe30 msgr2=0x7fc3d0149750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:54.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.114+0000 7fc3d7460700 1 -- 172.21.15.78:0/3981523007 shutdown_connections 2023-10-14T11:59:54.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.114+0000 7fc3d7460700 1 -- 172.21.15.78:0/3981523007 wait complete. 2023-10-14T11:59:54.127 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail testcephfs1 2023-10-14T11:59:54.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.247+0000 7f004be72700 1 Processor -- start 2023-10-14T11:59:54.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.248+0000 7f004be72700 1 -- start start 2023-10-14T11:59:54.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.248+0000 7f004be72700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f004413dbe0 0x7f004406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:54.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.248+0000 7f004be72700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f004406ae10 0x7f00440c09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:54.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.248+0000 7f004be72700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f004406ebe0 con 0x7f00440c4540 2023-10-14T11:59:54.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.248+0000 7f004be72700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f004406d330 con 0x7f004413dbe0 2023-10-14T11:59:54.255 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.248+0000 7f004be72700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00441558c0 con 0x7f004406ae10 2023-10-14T11:59:54.255 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.248+0000 7f0049c0e700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f004413dbe0 0x7f004406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:54.255 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.248+0000 7f004940d700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f00440c4540 0x7f00440c02e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51894/0 (socket says 172.21.15.78:51894) 2023-10-14T11:59:54.255 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.248+0000 7f004940d700 1 -- 172.21.15.78:0/1750565583 learned_addr learned my addr 172.21.15.78:0/1750565583 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:59:54.256 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.249+0000 7f004a40f700 1 --2- 172.21.15.78:0/1750565583 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f004406ae10 0x7f00440c09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:54.256 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.249+0000 7f0049c0e700 1 -- 172.21.15.78:0/1750565583 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f004406ae10 msgr2=0x7f00440c09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:54.256 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.249+0000 7f0049c0e700 1 --2- 172.21.15.78:0/1750565583 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f004406ae10 0x7f00440c09f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:54.256 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.249+0000 7f0049c0e700 1 -- 172.21.15.78:0/1750565583 >> v1:172.21.15.78:6789/0 conn(0x7f00440c4540 legacy=0x7f00440c02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:54.256 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.249+0000 7f0049c0e700 1 -- 172.21.15.78:0/1750565583 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0044154b00 con 0x7f004413dbe0 2023-10-14T11:59:54.257 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.249+0000 7f0048c0c700 1 -- 172.21.15.78:0/1750565583 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 557563522 0 0) 0x7f004406ebe0 con 0x7f00440c4540 2023-10-14T11:59:54.257 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.249+0000 7f0049c0e700 1 --2- 172.21.15.78:0/1750565583 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f004413dbe0 0x7f004406a8d0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f0034009840 tx=0x7f0034009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cea57060647021c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:54.257 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.249+0000 7f0048c0c700 1 -- 172.21.15.78:0/1750565583 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0034011830 con 0x7f004413dbe0 2023-10-14T11:59:54.257 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.249+0000 7f0048c0c700 1 -- 172.21.15.78:0/1750565583 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0034011990 con 0x7f004413dbe0 2023-10-14T11:59:54.257 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.249+0000 7f0048c0c700 1 -- 172.21.15.78:0/1750565583 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0034011d20 con 0x7f004413dbe0 2023-10-14T11:59:54.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.250+0000 7f004be72700 1 -- 172.21.15.78:0/1750565583 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f004413dbe0 msgr2=0x7f004406a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:54.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.250+0000 7f004be72700 1 --2- 172.21.15.78:0/1750565583 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f004413dbe0 0x7f004406a8d0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f0034009840 tx=0x7f0034009b50 comp rx=0 tx=0).stop 2023-10-14T11:59:54.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.250+0000 7f004be72700 1 -- 172.21.15.78:0/1750565583 shutdown_connections 2023-10-14T11:59:54.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.250+0000 7f004be72700 1 --2- 172.21.15.78:0/1750565583 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f004406ae10 0x7f00440c09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:54.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.250+0000 7f004be72700 1 --2- 172.21.15.78:0/1750565583 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f004413dbe0 0x7f004406a8d0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:54.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.250+0000 7f004be72700 1 -- 172.21.15.78:0/1750565583 >> 172.21.15.78:0/1750565583 conn(0x7f00440c3a50 msgr2=0x7f00440c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:54.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.250+0000 7f004be72700 1 -- 172.21.15.78:0/1750565583 shutdown_connections 2023-10-14T11:59:54.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.250+0000 7f004be72700 1 -- 172.21.15.78:0/1750565583 wait complete. 2023-10-14T11:59:54.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.251+0000 7f004be72700 1 Processor -- start 2023-10-14T11:59:54.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.251+0000 7f004be72700 1 -- start start 2023-10-14T11:59:54.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.251+0000 7f004be72700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f004406ae10 0x7f0044153940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:54.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.251+0000 7f004be72700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00440c4540 0x7f0044137a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:54.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.251+0000 7f004be72700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f004413dbe0 0x7f0044137f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:54.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.251+0000 7f004be72700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00440c40a0 con 0x7f00440c4540 2023-10-14T11:59:54.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.251+0000 7f004be72700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0044153e80 con 0x7f004413dbe0 2023-10-14T11:59:54.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.251+0000 7f004be72700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00441e9b30 con 0x7f004406ae10 2023-10-14T11:59:54.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.252+0000 7f004940d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00440c4540 0x7f0044137a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:54.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.252+0000 7f004940d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00440c4540 0x7f0044137a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47026/0 (socket says 172.21.15.78:47026) 2023-10-14T11:59:54.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.252+0000 7f004940d700 1 -- 172.21.15.78:0/3935519873 learned_addr learned my addr 172.21.15.78:0/3935519873 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:54.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.252+0000 7f0049c0e700 1 --2- 172.21.15.78:0/3935519873 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f004406ae10 0x7f0044153940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:54.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.252+0000 7f004a40f700 1 --2- 172.21.15.78:0/3935519873 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f004413dbe0 0x7f0044137f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:54.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.252+0000 7f004940d700 1 -- 172.21.15.78:0/3935519873 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f004406ae10 msgr2=0x7f0044153940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:54.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.252+0000 7f004940d700 1 --2- 172.21.15.78:0/3935519873 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f004406ae10 0x7f0044153940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:54.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.252+0000 7f004940d700 1 -- 172.21.15.78:0/3935519873 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f004413dbe0 msgr2=0x7f0044137f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:54.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.252+0000 7f004940d700 1 --2- 172.21.15.78:0/3935519873 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f004413dbe0 0x7f0044137f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:54.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.252+0000 7f004940d700 1 -- 172.21.15.78:0/3935519873 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00340094a0 con 0x7f00440c4540 2023-10-14T11:59:54.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.252+0000 7f004a40f700 1 --2- 172.21.15.78:0/3935519873 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f004413dbe0 0x7f0044137f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:59:54.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.252+0000 7f004940d700 1 --2- 172.21.15.78:0/3935519873 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00440c4540 0x7f0044137a10 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f004000c540 tx=0x7f004000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:54.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.252+0000 7f003a7fc700 1 -- 172.21.15.78:0/3935519873 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0040004c80 con 0x7f00440c4540 2023-10-14T11:59:54.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.252+0000 7f003a7fc700 1 -- 172.21.15.78:0/3935519873 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0040004de0 con 0x7f00440c4540 2023-10-14T11:59:54.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.252+0000 7f004be72700 1 -- 172.21.15.78:0/3935519873 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00441e9df0 con 0x7f00440c4540 2023-10-14T11:59:54.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.252+0000 7f003a7fc700 1 -- 172.21.15.78:0/3935519873 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0040007310 con 0x7f00440c4540 2023-10-14T11:59:54.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.252+0000 7f004be72700 1 -- 172.21.15.78:0/3935519873 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00441ea3b0 con 0x7f00440c4540 2023-10-14T11:59:54.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.253+0000 7f004be72700 1 -- 172.21.15.78:0/3935519873 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00440c5e20 con 0x7f00440c4540 2023-10-14T11:59:54.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.254+0000 7f003a7fc700 1 -- 172.21.15.78:0/3935519873 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0040007470 con 0x7f00440c4540 2023-10-14T11:59:54.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.254+0000 7f003a7fc700 1 --2- 172.21.15.78:0/3935519873 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f002806af00 0x7f002806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:54.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.254+0000 7f003a7fc700 1 -- 172.21.15.78:0/3935519873 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f0040089330 con 0x7f00440c4540 2023-10-14T11:59:54.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.254+0000 7f0049c0e700 1 --2- 172.21.15.78:0/3935519873 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f002806af00 0x7f002806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:54.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.254+0000 7f0049c0e700 1 --2- 172.21.15.78:0/3935519873 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f002806af00 0x7f002806d3c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f00340090b0 tx=0x7f003401c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:54.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.259+0000 7f003a7fc700 1 -- 172.21.15.78:0/3935519873 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0040054450 con 0x7f00440c4540 2023-10-14T11:59:54.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.494+0000 7f004be72700 1 -- 172.21.15.78:0/3935519873 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "testcephfs1"} v 0) v1 -- 0x7f0044074b80 con 0x7f00440c4540 2023-10-14T11:59:54.496 INFO:tasks.ceph.mon.a.smithi078.stderr:2023-10-14T11:59:54.495+0000 7f956c6f2700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-14T11:59:54.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.496+0000 7f003a7fc700 1 -- 172.21.15.78:0/3935519873 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "testcephfs1"}]=0 testcephfs1 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v18) v1 ==== 171+0+0 (secure 0 0 0) 0x7f0040057a70 con 0x7f00440c4540 2023-10-14T11:59:54.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.499+0000 7f004be72700 1 -- 172.21.15.78:0/3935519873 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f002806af00 msgr2=0x7f002806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:54.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.499+0000 7f004be72700 1 --2- 172.21.15.78:0/3935519873 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f002806af00 0x7f002806d3c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f00340090b0 tx=0x7f003401c040 comp rx=0 tx=0).stop 2023-10-14T11:59:54.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.499+0000 7f004be72700 1 -- 172.21.15.78:0/3935519873 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00440c4540 msgr2=0x7f0044137a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:54.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.499+0000 7f004be72700 1 --2- 172.21.15.78:0/3935519873 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00440c4540 0x7f0044137a10 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f004000c540 tx=0x7f004000c900 comp rx=0 tx=0).stop 2023-10-14T11:59:54.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.499+0000 7f004be72700 1 -- 172.21.15.78:0/3935519873 shutdown_connections 2023-10-14T11:59:54.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.499+0000 7f004be72700 1 --2- 172.21.15.78:0/3935519873 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f002806af00 0x7f002806d3c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:54.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.499+0000 7f004be72700 1 --2- 172.21.15.78:0/3935519873 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f004413dbe0 0x7f0044137f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:54.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.499+0000 7f004be72700 1 --2- 172.21.15.78:0/3935519873 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f004406ae10 0x7f0044153940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:54.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.499+0000 7f004be72700 1 --2- 172.21.15.78:0/3935519873 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00440c4540 0x7f0044137a10 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:54.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.499+0000 7f004be72700 1 -- 172.21.15.78:0/3935519873 >> 172.21.15.78:0/3935519873 conn(0x7f00440c3a50 msgr2=0x7f004413b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:54.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.500+0000 7f004be72700 1 -- 172.21.15.78:0/3935519873 shutdown_connections 2023-10-14T11:59:54.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.500+0000 7f004be72700 1 -- 172.21.15.78:0/3935519873 wait complete. 2023-10-14T11:59:54.504 INFO:tasks.ceph.mds.d.smithi195.stderr: -1492> 2023-10-14T11:59:05.148+0000 7f343474b700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-14T11:59:54.504 INFO:teuthology.orchestra.run.smithi078.stderr:testcephfs1 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-14T11:59:54.514 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm testcephfs1 --yes-i-really-mean-it 2023-10-14T11:59:54.528 INFO:tasks.ceph.mds.d.smithi195.stdout:starting mds.d at 2023-10-14T11:59:54.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.638+0000 7f490e485700 1 Processor -- start 2023-10-14T11:59:54.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.638+0000 7f490e485700 1 -- start start 2023-10-14T11:59:54.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.639+0000 7f490e485700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f49080c4540 0x7f49080c02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:54.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.639+0000 7f490e485700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f490806b500 0x7f49080c0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:54.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.639+0000 7f490e485700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f490806ebe0 con 0x7f490813dbe0 2023-10-14T11:59:54.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.639+0000 7f490e485700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f490806d330 con 0x7f490806b500 2023-10-14T11:59:54.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.639+0000 7f490e485700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49080c0d60 con 0x7f49080c4540 2023-10-14T11:59:54.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.639+0000 7f490ca22700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f490806b500 0x7f49080c0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:54.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.639+0000 7f490ca22700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f490806b500 0x7f49080c0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47036/0 (socket says 172.21.15.78:47036) 2023-10-14T11:59:54.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.639+0000 7f490ca22700 1 -- 172.21.15.78:0/1569313977 learned_addr learned my addr 172.21.15.78:0/1569313977 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:54.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.639+0000 7f49077fe700 1 --2- 172.21.15.78:0/1569313977 >> v2:172.21.15.195:3300/0 conn(0x7f49080c4540 0x7f49080c02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:54.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.640+0000 7f490ca22700 1 -- 172.21.15.78:0/1569313977 >> v2:172.21.15.195:3300/0 conn(0x7f49080c4540 msgr2=0x7f49080c02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:54.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.640+0000 7f490ca22700 1 --2- 172.21.15.78:0/1569313977 >> v2:172.21.15.195:3300/0 conn(0x7f49080c4540 0x7f49080c02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:54.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.640+0000 7f490ca22700 1 -- 172.21.15.78:0/1569313977 >> v1:172.21.15.195:6789/0 conn(0x7f490813dbe0 legacy=0x7f490806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:54.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.640+0000 7f490ca22700 1 -- 172.21.15.78:0/1569313977 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49081558c0 con 0x7f490806b500 2023-10-14T11:59:54.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.640+0000 7f49077fe700 1 --2- 172.21.15.78:0/1569313977 >> v2:172.21.15.195:3300/0 conn(0x7f49080c4540 0x7f49080c02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:59:54.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.640+0000 7f490ca22700 1 --2- 172.21.15.78:0/1569313977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f490806b500 0x7f49080c0820 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f490813e360 tx=0x7f490000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=714ec0f80d846c28 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:54.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.640+0000 7f4906ffd700 1 -- 172.21.15.78:0/1569313977 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490000e070 con 0x7f490806b500 2023-10-14T11:59:54.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.640+0000 7f4906ffd700 1 -- 172.21.15.78:0/1569313977 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f490000b9c0 con 0x7f490806b500 2023-10-14T11:59:54.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.640+0000 7f4906ffd700 1 -- 172.21.15.78:0/1569313977 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490000bd70 con 0x7f490806b500 2023-10-14T11:59:54.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.640+0000 7f490e485700 1 -- 172.21.15.78:0/1569313977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f490806b500 msgr2=0x7f49080c0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:54.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.640+0000 7f490e485700 1 --2- 172.21.15.78:0/1569313977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f490806b500 0x7f49080c0820 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f490813e360 tx=0x7f490000b330 comp rx=0 tx=0).stop 2023-10-14T11:59:54.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.640+0000 7f490e485700 1 -- 172.21.15.78:0/1569313977 shutdown_connections 2023-10-14T11:59:54.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.640+0000 7f490e485700 1 --2- 172.21.15.78:0/1569313977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f490806b500 0x7f49080c0820 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:54.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.640+0000 7f490e485700 1 --2- 172.21.15.78:0/1569313977 >> v2:172.21.15.195:3300/0 conn(0x7f49080c4540 0x7f49080c02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:54.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.640+0000 7f490e485700 1 -- 172.21.15.78:0/1569313977 >> 172.21.15.78:0/1569313977 conn(0x7f49080c3a50 msgr2=0x7f49080c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:54.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.640+0000 7f490e485700 1 -- 172.21.15.78:0/1569313977 shutdown_connections 2023-10-14T11:59:54.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.640+0000 7f490e485700 1 -- 172.21.15.78:0/1569313977 wait complete. 2023-10-14T11:59:54.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.641+0000 7f490e485700 1 Processor -- start 2023-10-14T11:59:54.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.642+0000 7f490e485700 1 -- start start 2023-10-14T11:59:54.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.642+0000 7f490e485700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f490806b500 0x7f49081560e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:54.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.642+0000 7f490e485700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f49080c4540 0x7f49080abe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:54.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.642+0000 7f490e485700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f490813dbe0 0x7f49081e9800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:54.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.642+0000 7f490e485700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49080c30e0 con 0x7f490806b500 2023-10-14T11:59:54.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.642+0000 7f490e485700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49081ecd70 con 0x7f49080c4540 2023-10-14T11:59:54.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.642+0000 7f490e485700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4908156910 con 0x7f490813dbe0 2023-10-14T11:59:54.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.642+0000 7f4907fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f490806b500 0x7f49081560e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:54.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.643+0000 7f4907fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f490806b500 0x7f49081560e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47038/0 (socket says 172.21.15.78:47038) 2023-10-14T11:59:54.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.643+0000 7f4907fff700 1 -- 172.21.15.78:0/1144116652 learned_addr learned my addr 172.21.15.78:0/1144116652 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:54.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.643+0000 7f49077fe700 1 --2- 172.21.15.78:0/1144116652 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f49080c4540 0x7f49080abe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:54.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.643+0000 7f490ca22700 1 --2- 172.21.15.78:0/1144116652 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f490813dbe0 0x7f49081e9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:54.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.643+0000 7f4907fff700 1 -- 172.21.15.78:0/1144116652 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f490813dbe0 msgr2=0x7f49081e9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:54.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.643+0000 7f4907fff700 1 --2- 172.21.15.78:0/1144116652 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f490813dbe0 0x7f49081e9800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:54.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.643+0000 7f4907fff700 1 -- 172.21.15.78:0/1144116652 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f49080c4540 msgr2=0x7f49080abe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:54.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.643+0000 7f4907fff700 1 --2- 172.21.15.78:0/1144116652 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f49080c4540 0x7f49080abe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:54.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.643+0000 7f4907fff700 1 -- 172.21.15.78:0/1144116652 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4900009b30 con 0x7f490806b500 2023-10-14T11:59:54.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.643+0000 7f490ca22700 1 --2- 172.21.15.78:0/1144116652 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f490813dbe0 0x7f49081e9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:59:54.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.643+0000 7f4907fff700 1 --2- 172.21.15.78:0/1144116652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f490806b500 0x7f49081560e0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f48f800c450 tx=0x7f48f800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:54.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.643+0000 7f4904ff9700 1 -- 172.21.15.78:0/1144116652 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f80052d0 con 0x7f490806b500 2023-10-14T11:59:54.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.643+0000 7f4904ff9700 1 -- 172.21.15.78:0/1144116652 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48f8005430 con 0x7f490806b500 2023-10-14T11:59:54.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.643+0000 7f490e485700 1 -- 172.21.15.78:0/1144116652 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49081ed010 con 0x7f490806b500 2023-10-14T11:59:54.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.643+0000 7f4904ff9700 1 -- 172.21.15.78:0/1144116652 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f80038c0 con 0x7f490806b500 2023-10-14T11:59:54.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.643+0000 7f490e485700 1 -- 172.21.15.78:0/1144116652 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49081ed5d0 con 0x7f490806b500 2023-10-14T11:59:54.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.644+0000 7f490e485700 1 -- 172.21.15.78:0/1144116652 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4908074b80 con 0x7f490806b500 2023-10-14T11:59:54.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.644+0000 7f4904ff9700 1 -- 172.21.15.78:0/1144116652 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f48f8003a20 con 0x7f490806b500 2023-10-14T11:59:54.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.645+0000 7f4904ff9700 1 --2- 172.21.15.78:0/1144116652 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f48f406af00 0x7f48f406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:54.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.645+0000 7f4904ff9700 1 -- 172.21.15.78:0/1144116652 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5815+0+0 (secure 0 0 0) 0x7f48f8088910 con 0x7f490806b500 2023-10-14T11:59:54.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.645+0000 7f49077fe700 1 --2- 172.21.15.78:0/1144116652 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f48f406af00 0x7f48f406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:54.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.645+0000 7f49077fe700 1 --2- 172.21.15.78:0/1144116652 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f48f406af00 0x7f48f406d3c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f48fc006f10 tx=0x7f48fc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:54.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.649+0000 7f4904ff9700 1 -- 172.21.15.78:0/1144116652 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f48f80539e0 con 0x7f490806b500 2023-10-14T11:59:54.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:54.883+0000 7f490e485700 1 -- 172.21.15.78:0/1144116652 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "testcephfs1", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f4908154ab0 con 0x7f490806b500 2023-10-14T11:59:55.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.500+0000 7f4904ff9700 1 -- 172.21.15.78:0/1144116652 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "testcephfs1", "yes_i_really_mean_it": true}]=0 v19) v1 ==== 109+0+0 (secure 0 0 0) 0x7f48f80170b0 con 0x7f490806b500 2023-10-14T11:59:55.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.502+0000 7f490e485700 1 -- 172.21.15.78:0/1144116652 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f48f406af00 msgr2=0x7f48f406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:55.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.502+0000 7f490e485700 1 --2- 172.21.15.78:0/1144116652 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f48f406af00 0x7f48f406d3c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f48fc006f10 tx=0x7f48fc008040 comp rx=0 tx=0).stop 2023-10-14T11:59:55.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.502+0000 7f490e485700 1 -- 172.21.15.78:0/1144116652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f490806b500 msgr2=0x7f49081560e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:55.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.502+0000 7f490e485700 1 --2- 172.21.15.78:0/1144116652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f490806b500 0x7f49081560e0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f48f800c450 tx=0x7f48f800c760 comp rx=0 tx=0).stop 2023-10-14T11:59:55.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.502+0000 7f490e485700 1 -- 172.21.15.78:0/1144116652 shutdown_connections 2023-10-14T11:59:55.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.502+0000 7f490e485700 1 --2- 172.21.15.78:0/1144116652 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f490813dbe0 0x7f49081e9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:55.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.502+0000 7f490e485700 1 --2- 172.21.15.78:0/1144116652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f490806b500 0x7f49081560e0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:55.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.502+0000 7f490e485700 1 --2- 172.21.15.78:0/1144116652 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f48f406af00 0x7f48f406d3c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:55.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.502+0000 7f490e485700 1 --2- 172.21.15.78:0/1144116652 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f49080c4540 0x7f49080abe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:55.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.502+0000 7f490e485700 1 -- 172.21.15.78:0/1144116652 >> 172.21.15.78:0/1144116652 conn(0x7f49080c3a50 msgr2=0x7f49080c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:55.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.503+0000 7f490e485700 1 -- 172.21.15.78:0/1144116652 shutdown_connections 2023-10-14T11:59:55.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.503+0000 7f490e485700 1 -- 172.21.15.78:0/1144116652 wait complete. 2023-10-14T11:59:55.517 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm testcephfs1_metadata testcephfs1_metadata --yes-i-really-really-mean-it 2023-10-14T11:59:55.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.645+0000 7f73fee33700 1 Processor -- start 2023-10-14T11:59:55.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.645+0000 7f73fee33700 1 -- start start 2023-10-14T11:59:55.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.646+0000 7f73fee33700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f73f806ff10 0x7f73f80c5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:55.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.646+0000 7f73fee33700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73f80c5070 con 0x7f73f80ad7c0 2023-10-14T11:59:55.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.646+0000 7f73fee33700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73f81543d0 con 0x7f73f806f330 2023-10-14T11:59:55.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.646+0000 7f73fee33700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73f80c4840 con 0x7f73f806ff10 2023-10-14T11:59:55.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.646+0000 7f73fcbcf700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f73f80ad7c0 0x7f73f8153cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51922/0 (socket says 172.21.15.78:51922) 2023-10-14T11:59:55.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.646+0000 7f73fcbcf700 1 -- 172.21.15.78:0/2179345945 learned_addr learned my addr 172.21.15.78:0/2179345945 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:59:55.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.646+0000 7f73f7fff700 1 --2- 172.21.15.78:0/2179345945 >> v2:172.21.15.195:3300/0 conn(0x7f73f806ff10 0x7f73f80c5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:55.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.646+0000 7f73f77fe700 1 -- 172.21.15.78:0/2179345945 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3764246334 0 0) 0x7f73f80c5070 con 0x7f73f80ad7c0 2023-10-14T11:59:55.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.646+0000 7f73f77fe700 1 -- 172.21.15.78:0/2179345945 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73e80035b0 con 0x7f73f80ad7c0 2023-10-14T11:59:55.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.647+0000 7f73f77fe700 1 -- 172.21.15.78:0/2179345945 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 43149621 0 0) 0x7f73e80035b0 con 0x7f73f80ad7c0 2023-10-14T11:59:55.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.647+0000 7f73f77fe700 1 -- 172.21.15.78:0/2179345945 >> v2:172.21.15.195:3300/0 conn(0x7f73f806ff10 msgr2=0x7f73f80c5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:55.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.647+0000 7f73f77fe700 1 --2- 172.21.15.78:0/2179345945 >> v2:172.21.15.195:3300/0 conn(0x7f73f806ff10 0x7f73f80c5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:55.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.647+0000 7f73f77fe700 1 -- 172.21.15.78:0/2179345945 >> v1:172.21.15.195:6789/0 conn(0x7f73f806f330 legacy=0x7f73f8155dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:55.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.647+0000 7f73f77fe700 1 -- 172.21.15.78:0/2179345945 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f806f8f0 con 0x7f73f80ad7c0 2023-10-14T11:59:55.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.647+0000 7f73f77fe700 1 -- 172.21.15.78:0/2179345945 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441799917 0 0) 0x7f73ec001670 con 0x7f73f80ad7c0 2023-10-14T11:59:55.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.647+0000 7f73f77fe700 1 -- 172.21.15.78:0/2179345945 >> v1:172.21.15.78:6789/0 conn(0x7f73f80ad7c0 legacy=0x7f73f8153cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:55.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.647+0000 7f73f77fe700 1 --2- 172.21.15.78:0/2179345945 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f73e8003a00 0x7f73e8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:55.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.647+0000 7f73f77fe700 1 --2- 172.21.15.78:0/2179345945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73e8004850 0x7f73e8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:55.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.647+0000 7f73f77fe700 1 --2- 172.21.15.78:0/2179345945 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f73e8005210 0x7f73e800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:55.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.647+0000 7f73f77fe700 1 -- 172.21.15.78:0/2179345945 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f80c4840 con 0x7f73e8004850 2023-10-14T11:59:55.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.647+0000 7f73f77fe700 1 -- 172.21.15.78:0/2179345945 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e800c0b0 con 0x7f73e8003a00 2023-10-14T11:59:55.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.647+0000 7f73f77fe700 1 -- 172.21.15.78:0/2179345945 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e800c270 con 0x7f73e8005210 2023-10-14T11:59:55.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.647+0000 7f73f7fff700 1 --2- 172.21.15.78:0/2179345945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73e8004850 0x7f73e8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:55.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.647+0000 7f73fcbcf700 1 --2- 172.21.15.78:0/2179345945 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f73e8003a00 0x7f73e8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:55.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.647+0000 7f73fd3d0700 1 --2- 172.21.15.78:0/2179345945 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f73e8005210 0x7f73e800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:55.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.647+0000 7f73f7fff700 1 -- 172.21.15.78:0/2179345945 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f73e8005210 msgr2=0x7f73e800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:55.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.647+0000 7f73f7fff700 1 --2- 172.21.15.78:0/2179345945 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f73e8005210 0x7f73e800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:55.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.647+0000 7f73f7fff700 1 -- 172.21.15.78:0/2179345945 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f73e8003a00 msgr2=0x7f73e8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:55.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.647+0000 7f73f7fff700 1 --2- 172.21.15.78:0/2179345945 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f73e8003a00 0x7f73e8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:55.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.647+0000 7f73f7fff700 1 -- 172.21.15.78:0/2179345945 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e80035b0 con 0x7f73e8004850 2023-10-14T11:59:55.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.647+0000 7f73fd3d0700 1 -- 172.21.15.78:0/2179345945 reap_dead start 2023-10-14T11:59:55.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.647+0000 7f73f7fff700 1 --2- 172.21.15.78:0/2179345945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73e8004850 0x7f73e8004cd0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f73e4009b60 tx=0x7f73e8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=299c2ea0730249e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:55.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.647+0000 7f73f77fe700 1 -- 172.21.15.78:0/2179345945 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e400e6c0 con 0x7f73e8004850 2023-10-14T11:59:55.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.647+0000 7f73f77fe700 1 -- 172.21.15.78:0/2179345945 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73e400e820 con 0x7f73e8004850 2023-10-14T11:59:55.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.647+0000 7f73f77fe700 1 -- 172.21.15.78:0/2179345945 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e400eb90 con 0x7f73e8004850 2023-10-14T11:59:55.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.648+0000 7f73fee33700 1 -- 172.21.15.78:0/2179345945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73e8004850 msgr2=0x7f73e8004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:55.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.648+0000 7f73fee33700 1 --2- 172.21.15.78:0/2179345945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73e8004850 0x7f73e8004cd0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f73e4009b60 tx=0x7f73e8005880 comp rx=0 tx=0).stop 2023-10-14T11:59:55.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.648+0000 7f73fee33700 1 -- 172.21.15.78:0/2179345945 shutdown_connections 2023-10-14T11:59:55.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.648+0000 7f73fee33700 1 --2- 172.21.15.78:0/2179345945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73e8004850 0x7f73e8004cd0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:55.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.648+0000 7f73fee33700 1 -- 172.21.15.78:0/2179345945 >> 172.21.15.78:0/2179345945 conn(0x7f73f8072770 msgr2=0x7f73f80733e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:55.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.648+0000 7f73fee33700 1 -- 172.21.15.78:0/2179345945 shutdown_connections 2023-10-14T11:59:55.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.648+0000 7f73fee33700 1 -- 172.21.15.78:0/2179345945 wait complete. 2023-10-14T11:59:55.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.649+0000 7f73fee33700 1 Processor -- start 2023-10-14T11:59:55.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.649+0000 7f73fee33700 1 -- start start 2023-10-14T11:59:55.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.650+0000 7f73fee33700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f73e8004850 0x7f73f81eb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:55.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.650+0000 7f73fee33700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73f80c7df0 0x7f73f81ea220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:55.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.650+0000 7f73fee33700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f73f81ea760 0x7f73f81e7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:55.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.650+0000 7f73fee33700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f81ebd00 con 0x7f73f80c7df0 2023-10-14T11:59:55.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.650+0000 7f73fee33700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f81e7920 con 0x7f73e8004850 2023-10-14T11:59:55.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.650+0000 7f73fee33700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73f80c1650 con 0x7f73f81ea760 2023-10-14T11:59:55.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.650+0000 7f73f7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73f80c7df0 0x7f73f81ea220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:55.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.650+0000 7f73f7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73f80c7df0 0x7f73f81ea220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47060/0 (socket says 172.21.15.78:47060) 2023-10-14T11:59:55.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.650+0000 7f73f7fff700 1 -- 172.21.15.78:0/3461924739 learned_addr learned my addr 172.21.15.78:0/3461924739 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:55.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.650+0000 7f73fd3d0700 1 --2- 172.21.15.78:0/3461924739 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f73f81ea760 0x7f73f81e7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:55.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.650+0000 7f73fcbcf700 1 --2- 172.21.15.78:0/3461924739 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f73e8004850 0x7f73f81eb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:55.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.650+0000 7f73f7fff700 1 -- 172.21.15.78:0/3461924739 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f73f81ea760 msgr2=0x7f73f81e7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:55.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.650+0000 7f73f7fff700 1 --2- 172.21.15.78:0/3461924739 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f73f81ea760 0x7f73f81e7e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:55.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.650+0000 7f73f7fff700 1 -- 172.21.15.78:0/3461924739 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f73e8004850 msgr2=0x7f73f81eb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:55.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.650+0000 7f73f7fff700 1 --2- 172.21.15.78:0/3461924739 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f73e8004850 0x7f73f81eb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:55.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.650+0000 7f73f7fff700 1 -- 172.21.15.78:0/3461924739 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f80c1930 con 0x7f73f80c7df0 2023-10-14T11:59:55.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.651+0000 7f73f7fff700 1 --2- 172.21.15.78:0/3461924739 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73f80c7df0 0x7f73f81ea220 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f73e800c080 tx=0x7f73e40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:55.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.651+0000 7f73f57fa700 1 -- 172.21.15.78:0/3461924739 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e400e6c0 con 0x7f73f80c7df0 2023-10-14T11:59:55.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.651+0000 7f73f57fa700 1 -- 172.21.15.78:0/3461924739 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73e4003ad0 con 0x7f73f80c7df0 2023-10-14T11:59:55.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.651+0000 7f73fee33700 1 -- 172.21.15.78:0/3461924739 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73f80c1c10 con 0x7f73f80c7df0 2023-10-14T11:59:55.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.651+0000 7f73f57fa700 1 -- 172.21.15.78:0/3461924739 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e4003e80 con 0x7f73f80c7df0 2023-10-14T11:59:55.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.651+0000 7f73fee33700 1 -- 172.21.15.78:0/3461924739 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73f81eabe0 con 0x7f73f80c7df0 2023-10-14T11:59:55.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.652+0000 7f73fee33700 1 -- 172.21.15.78:0/3461924739 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73f8074b80 con 0x7f73f80c7df0 2023-10-14T11:59:55.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.652+0000 7f73f57fa700 1 -- 172.21.15.78:0/3461924739 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f73e4013470 con 0x7f73f80c7df0 2023-10-14T11:59:55.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.653+0000 7f73f57fa700 1 --2- 172.21.15.78:0/3461924739 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f73dc06af00 0x7f73dc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:55.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.653+0000 7f73fcbcf700 1 --2- 172.21.15.78:0/3461924739 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f73dc06af00 0x7f73dc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:55.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.653+0000 7f73f57fa700 1 -- 172.21.15.78:0/3461924739 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5815+0+0 (secure 0 0 0) 0x7f73e408ae60 con 0x7f73f80c7df0 2023-10-14T11:59:55.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.653+0000 7f73fcbcf700 1 --2- 172.21.15.78:0/3461924739 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f73dc06af00 0x7f73dc06d3c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f73ec007cd0 tx=0x7f73ec009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:55.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.657+0000 7f73f57fa700 1 -- 172.21.15.78:0/3461924739 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f73e4055fd0 con 0x7f73f80c7df0 2023-10-14T11:59:55.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:55.847+0000 7f73fee33700 1 -- 172.21.15.78:0/3461924739 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "testcephfs1_metadata", "pool2": "testcephfs1_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f73f80c2260 con 0x7f73f80c7df0 2023-10-14T11:59:56.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.501+0000 7f73f57fa700 1 -- 172.21.15.78:0/3461924739 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "testcephfs1_metadata", "pool2": "testcephfs1_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'testcephfs1_metadata' removed v34) v1 ==== 196+0+0 (secure 0 0 0) 0x7f73e40170e0 con 0x7f73f80c7df0 2023-10-14T11:59:56.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.503+0000 7f73fee33700 1 -- 172.21.15.78:0/3461924739 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f73dc06af00 msgr2=0x7f73dc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:56.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.503+0000 7f73fee33700 1 --2- 172.21.15.78:0/3461924739 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f73dc06af00 0x7f73dc06d3c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f73ec007cd0 tx=0x7f73ec009040 comp rx=0 tx=0).stop 2023-10-14T11:59:56.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.503+0000 7f73fee33700 1 -- 172.21.15.78:0/3461924739 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73f80c7df0 msgr2=0x7f73f81ea220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:56.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.503+0000 7f73fee33700 1 --2- 172.21.15.78:0/3461924739 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73f80c7df0 0x7f73f81ea220 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f73e800c080 tx=0x7f73e40047c0 comp rx=0 tx=0).stop 2023-10-14T11:59:56.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.504+0000 7f73fee33700 1 -- 172.21.15.78:0/3461924739 shutdown_connections 2023-10-14T11:59:56.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.504+0000 7f73fee33700 1 --2- 172.21.15.78:0/3461924739 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f73f81ea760 0x7f73f81e7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:56.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.504+0000 7f73fee33700 1 --2- 172.21.15.78:0/3461924739 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f73dc06af00 0x7f73dc06d3c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:56.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.504+0000 7f73fee33700 1 --2- 172.21.15.78:0/3461924739 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f73e8004850 0x7f73f81eb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:56.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.504+0000 7f73fee33700 1 --2- 172.21.15.78:0/3461924739 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73f80c7df0 0x7f73f81ea220 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:56.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.504+0000 7f73fee33700 1 -- 172.21.15.78:0/3461924739 >> 172.21.15.78:0/3461924739 conn(0x7f73f8072770 msgr2=0x7f73f80ac7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:56.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.504+0000 7f73fee33700 1 -- 172.21.15.78:0/3461924739 shutdown_connections 2023-10-14T11:59:56.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.504+0000 7f73fee33700 1 -- 172.21.15.78:0/3461924739 wait complete. 2023-10-14T11:59:56.507 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'testcephfs1_metadata' removed 2023-10-14T11:59:56.521 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm testcephfs1_data testcephfs1_data --yes-i-really-really-mean-it 2023-10-14T11:59:56.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.647+0000 7f42aaa17700 1 Processor -- start 2023-10-14T11:59:56.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.647+0000 7f42aaa17700 1 -- start start 2023-10-14T11:59:56.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.647+0000 7f42aaa17700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f42a40c3e70 0x7f42a406d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:56.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.647+0000 7f42aaa17700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f42a406f820 0x7f42a40c4250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:56.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.647+0000 7f42aaa17700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42a4157ee0 con 0x7f42a406b9e0 2023-10-14T11:59:56.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.647+0000 7f42aaa17700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42a40c2fe0 con 0x7f42a406f820 2023-10-14T11:59:56.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.647+0000 7f42aaa17700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42a41581c0 con 0x7f42a40c3e70 2023-10-14T11:59:56.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.648+0000 7f42a37fe700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f42a406b9e0 0x7f42a4158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51942/0 (socket says 172.21.15.78:51942) 2023-10-14T11:59:56.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.648+0000 7f42a8fb4700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f42a406f820 0x7f42a40c4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:56.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.648+0000 7f42a37fe700 1 -- 172.21.15.78:0/2996554013 learned_addr learned my addr 172.21.15.78:0/2996554013 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:59:56.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.648+0000 7f42a8fb4700 1 -- 172.21.15.78:0/2996554013 >> v2:172.21.15.195:3300/0 conn(0x7f42a40c3e70 msgr2=0x7f42a406d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:56.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.648+0000 7f42a8fb4700 1 --2- 172.21.15.78:0/2996554013 >> v2:172.21.15.195:3300/0 conn(0x7f42a40c3e70 0x7f42a406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:56.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.648+0000 7f42a8fb4700 1 -- 172.21.15.78:0/2996554013 >> v1:172.21.15.78:6789/0 conn(0x7f42a406b9e0 legacy=0x7f42a4158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:56.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.648+0000 7f42a8fb4700 1 -- 172.21.15.78:0/2996554013 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a40702f0 con 0x7f42a406f820 2023-10-14T11:59:56.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.648+0000 7f42a2ffd700 1 -- 172.21.15.78:0/2996554013 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2764049843 0 0) 0x7f42a4157ee0 con 0x7f42a406b9e0 2023-10-14T11:59:56.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.648+0000 7f42a8fb4700 1 --2- 172.21.15.78:0/2996554013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42a406f820 0x7f42a40c4250 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f42a40c49c0 tx=0x7f429c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e5adbfb84d60c75 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:56.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.648+0000 7f42a2ffd700 1 -- 172.21.15.78:0/2996554013 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f429c00e070 con 0x7f42a406f820 2023-10-14T11:59:56.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.648+0000 7f42a2ffd700 1 -- 172.21.15.78:0/2996554013 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f429c00bbd0 con 0x7f42a406f820 2023-10-14T11:59:56.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.648+0000 7f42a2ffd700 1 -- 172.21.15.78:0/2996554013 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f429c012430 con 0x7f42a406f820 2023-10-14T11:59:56.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.649+0000 7f42aaa17700 1 -- 172.21.15.78:0/2996554013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42a406f820 msgr2=0x7f42a40c4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:56.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.649+0000 7f42aaa17700 1 --2- 172.21.15.78:0/2996554013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42a406f820 0x7f42a40c4250 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f42a40c49c0 tx=0x7f429c00b330 comp rx=0 tx=0).stop 2023-10-14T11:59:56.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.649+0000 7f42aaa17700 1 -- 172.21.15.78:0/2996554013 shutdown_connections 2023-10-14T11:59:56.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.649+0000 7f42aaa17700 1 --2- 172.21.15.78:0/2996554013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42a406f820 0x7f42a40c4250 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:56.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.649+0000 7f42aaa17700 1 --2- 172.21.15.78:0/2996554013 >> v2:172.21.15.195:3300/0 conn(0x7f42a40c3e70 0x7f42a406d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:56.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.649+0000 7f42aaa17700 1 -- 172.21.15.78:0/2996554013 >> 172.21.15.78:0/2996554013 conn(0x7f42a4072770 msgr2=0x7f42a40733e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:56.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.649+0000 7f42aaa17700 1 -- 172.21.15.78:0/2996554013 shutdown_connections 2023-10-14T11:59:56.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.649+0000 7f42aaa17700 1 -- 172.21.15.78:0/2996554013 wait complete. 2023-10-14T11:59:56.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.650+0000 7f42aaa17700 1 Processor -- start 2023-10-14T11:59:56.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.650+0000 7f42aaa17700 1 -- start start 2023-10-14T11:59:56.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.651+0000 7f42aaa17700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f42a406b9e0 0x7f42a40a9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:56.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.651+0000 7f42aaa17700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42a406f820 0x7f42a41ea290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:56.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.651+0000 7f42aaa17700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f42a40c3e70 0x7f42a406c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:56.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.651+0000 7f42aaa17700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a41ea7d0 con 0x7f42a406f820 2023-10-14T11:59:56.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.651+0000 7f42aaa17700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a4152f60 con 0x7f42a406b9e0 2023-10-14T11:59:56.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.651+0000 7f42aaa17700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42a41576c0 con 0x7f42a40c3e70 2023-10-14T11:59:56.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.651+0000 7f42a37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42a406f820 0x7f42a41ea290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:56.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.651+0000 7f42a37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42a406f820 0x7f42a41ea290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47074/0 (socket says 172.21.15.78:47074) 2023-10-14T11:59:56.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.651+0000 7f42a37fe700 1 -- 172.21.15.78:0/2482547455 learned_addr learned my addr 172.21.15.78:0/2482547455 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:56.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.651+0000 7f42a3fff700 1 --2- 172.21.15.78:0/2482547455 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f42a406b9e0 0x7f42a40a9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:56.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.651+0000 7f42a8fb4700 1 --2- 172.21.15.78:0/2482547455 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f42a40c3e70 0x7f42a406c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:56.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.651+0000 7f42a37fe700 1 -- 172.21.15.78:0/2482547455 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f42a40c3e70 msgr2=0x7f42a406c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:56.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.651+0000 7f42a37fe700 1 --2- 172.21.15.78:0/2482547455 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f42a40c3e70 0x7f42a406c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:56.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.651+0000 7f42a37fe700 1 -- 172.21.15.78:0/2482547455 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f42a406b9e0 msgr2=0x7f42a40a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:56.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.651+0000 7f42a37fe700 1 --2- 172.21.15.78:0/2482547455 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f42a406b9e0 0x7f42a40a9850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:56.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.651+0000 7f42a37fe700 1 -- 172.21.15.78:0/2482547455 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f429c009a70 con 0x7f42a406f820 2023-10-14T11:59:56.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.651+0000 7f42a37fe700 1 --2- 172.21.15.78:0/2482547455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42a406f820 0x7f42a41ea290 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f42980011e0 tx=0x7f429800cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:56.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.651+0000 7f42a0ff9700 1 -- 172.21.15.78:0/2482547455 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4298004c80 con 0x7f42a406f820 2023-10-14T11:59:56.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.651+0000 7f42a0ff9700 1 -- 172.21.15.78:0/2482547455 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4298004e00 con 0x7f42a406f820 2023-10-14T11:59:56.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.651+0000 7f42aaa17700 1 -- 172.21.15.78:0/2482547455 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42a41579a0 con 0x7f42a406f820 2023-10-14T11:59:56.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.651+0000 7f42a0ff9700 1 -- 172.21.15.78:0/2482547455 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4298007310 con 0x7f42a406f820 2023-10-14T11:59:56.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.651+0000 7f42aaa17700 1 -- 172.21.15.78:0/2482547455 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42a41ecda0 con 0x7f42a406f820 2023-10-14T11:59:56.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.652+0000 7f42aaa17700 1 -- 172.21.15.78:0/2482547455 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42a4074b80 con 0x7f42a406f820 2023-10-14T11:59:56.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.653+0000 7f42a0ff9700 1 -- 172.21.15.78:0/2482547455 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4298007470 con 0x7f42a406f820 2023-10-14T11:59:56.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.653+0000 7f42a0ff9700 1 --2- 172.21.15.78:0/2482547455 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f429006af00 0x7f429006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:56.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.653+0000 7f42a0ff9700 1 -- 172.21.15.78:0/2482547455 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5367+0+0 (secure 0 0 0) 0x7f4298089210 con 0x7f42a406f820 2023-10-14T11:59:56.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.653+0000 7f42a3fff700 1 --2- 172.21.15.78:0/2482547455 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f429006af00 0x7f429006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:56.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.654+0000 7f42a3fff700 1 --2- 172.21.15.78:0/2482547455 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f429006af00 0x7f429006d3c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4294006fd0 tx=0x7f4294008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:56.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.658+0000 7f42a0ff9700 1 -- 172.21.15.78:0/2482547455 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f42980544a0 con 0x7f42a406f820 2023-10-14T11:59:56.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:56.848+0000 7f42aaa17700 1 -- 172.21.15.78:0/2482547455 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "testcephfs1_data", "pool2": "testcephfs1_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f42a40adfe0 con 0x7f42a406f820 2023-10-14T11:59:57.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.507+0000 7f42a0ff9700 1 -- 172.21.15.78:0/2482547455 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "testcephfs1_data", "pool2": "testcephfs1_data", "yes_i_really_really_mean_it": true}]=0 pool 'testcephfs1_data' removed v35) v1 ==== 184+0+0 (secure 0 0 0) 0x7f4298057ac0 con 0x7f42a406f820 2023-10-14T11:59:57.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.510+0000 7f42aaa17700 1 -- 172.21.15.78:0/2482547455 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f429006af00 msgr2=0x7f429006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:57.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.510+0000 7f42aaa17700 1 --2- 172.21.15.78:0/2482547455 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f429006af00 0x7f429006d3c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4294006fd0 tx=0x7f4294008040 comp rx=0 tx=0).stop 2023-10-14T11:59:57.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.510+0000 7f42aaa17700 1 -- 172.21.15.78:0/2482547455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42a406f820 msgr2=0x7f42a41ea290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:57.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.510+0000 7f42aaa17700 1 --2- 172.21.15.78:0/2482547455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42a406f820 0x7f42a41ea290 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f42980011e0 tx=0x7f429800cae0 comp rx=0 tx=0).stop 2023-10-14T11:59:57.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.510+0000 7f42aaa17700 1 -- 172.21.15.78:0/2482547455 shutdown_connections 2023-10-14T11:59:57.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.510+0000 7f42aaa17700 1 --2- 172.21.15.78:0/2482547455 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f42a40c3e70 0x7f42a406c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:57.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.510+0000 7f42aaa17700 1 --2- 172.21.15.78:0/2482547455 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f429006af00 0x7f429006d3c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:57.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.510+0000 7f42aaa17700 1 --2- 172.21.15.78:0/2482547455 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f42a406b9e0 0x7f42a40a9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:57.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.510+0000 7f42aaa17700 1 --2- 172.21.15.78:0/2482547455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42a406f820 0x7f42a41ea290 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:57.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.510+0000 7f42aaa17700 1 -- 172.21.15.78:0/2482547455 >> 172.21.15.78:0/2482547455 conn(0x7f42a4072770 msgr2=0x7f42a40733e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:57.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.510+0000 7f42aaa17700 1 -- 172.21.15.78:0/2482547455 shutdown_connections 2023-10-14T11:59:57.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.510+0000 7f42aaa17700 1 -- 172.21.15.78:0/2482547455 wait complete. 2023-10-14T11:59:57.514 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'testcephfs1_data' removed 2023-10-14T11:59:57.527 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.b mds_join_fs 2023-10-14T11:59:57.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.650+0000 7f2420f58700 1 Processor -- start 2023-10-14T11:59:57.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.650+0000 7f2420f58700 1 -- start start 2023-10-14T11:59:57.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.651+0000 7f2420f58700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f241c0a70c0 0x7f241c154960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:57.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.651+0000 7f2420f58700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f241c154ea0 0x7f241c15d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:57.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.651+0000 7f2420f58700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f241c1537b0 con 0x7f241c1533d0 2023-10-14T11:59:57.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.651+0000 7f2420f58700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f241c0c6fb0 con 0x7f241c154ea0 2023-10-14T11:59:57.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.651+0000 7f2420f58700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f241c0c7f30 con 0x7f241c0a70c0 2023-10-14T11:59:57.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.651+0000 7f241a59c700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f241c1533d0 0x7f241c0c8680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51956/0 (socket says 172.21.15.78:51956) 2023-10-14T11:59:57.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.651+0000 7f241a59c700 1 -- 172.21.15.78:0/2119114322 learned_addr learned my addr 172.21.15.78:0/2119114322 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T11:59:57.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.651+0000 7f241ad9d700 1 --2- 172.21.15.78:0/2119114322 >> v2:172.21.15.195:3300/0 conn(0x7f241c154ea0 0x7f241c15d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:57.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.651+0000 7f2419d9b700 1 --2- 172.21.15.78:0/2119114322 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f241c0a70c0 0x7f241c154960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:57.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.652+0000 7f241959a700 1 -- 172.21.15.78:0/2119114322 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1439327584 0 0) 0x7f241c1537b0 con 0x7f241c1533d0 2023-10-14T11:59:57.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.652+0000 7f241959a700 1 -- 172.21.15.78:0/2119114322 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24040035b0 con 0x7f241c1533d0 2023-10-14T11:59:57.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.652+0000 7f241959a700 1 -- 172.21.15.78:0/2119114322 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1728103256 0 0) 0x7f24040035b0 con 0x7f241c1533d0 2023-10-14T11:59:57.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.652+0000 7f241959a700 1 -- 172.21.15.78:0/2119114322 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f241c0a70c0 msgr2=0x7f241c154960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:57.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.652+0000 7f241959a700 1 --2- 172.21.15.78:0/2119114322 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f241c0a70c0 0x7f241c154960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:57.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.652+0000 7f241959a700 1 -- 172.21.15.78:0/2119114322 >> v2:172.21.15.195:3300/0 conn(0x7f241c154ea0 msgr2=0x7f241c15d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:57.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.652+0000 7f241959a700 1 --2- 172.21.15.78:0/2119114322 >> v2:172.21.15.195:3300/0 conn(0x7f241c154ea0 0x7f241c15d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:57.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.652+0000 7f241959a700 1 -- 172.21.15.78:0/2119114322 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f241c0c7c50 con 0x7f241c1533d0 2023-10-14T11:59:57.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.652+0000 7f241959a700 1 -- 172.21.15.78:0/2119114322 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2441799917 0 0) 0x7f2410001670 con 0x7f241c1533d0 2023-10-14T11:59:57.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.652+0000 7f2419d9b700 1 --2- 172.21.15.78:0/2119114322 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f241c0a70c0 0x7f241c154960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T11:59:57.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.652+0000 7f241959a700 1 -- 172.21.15.78:0/2119114322 >> v1:172.21.15.78:6789/0 conn(0x7f241c1533d0 legacy=0x7f241c0c8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:57.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.652+0000 7f241959a700 1 --2- 172.21.15.78:0/2119114322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2404003a00 0x7f2404003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:57.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.652+0000 7f241959a700 1 --2- 172.21.15.78:0/2119114322 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f241c0a70c0 0x7f2404004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:57.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.652+0000 7f241959a700 1 --2- 172.21.15.78:0/2119114322 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2404004d90 0x7f2404005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:57.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.652+0000 7f241959a700 1 -- 172.21.15.78:0/2119114322 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241c0c6fb0 con 0x7f2404003a00 2023-10-14T11:59:57.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.652+0000 7f241959a700 1 -- 172.21.15.78:0/2119114322 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241c0c7f30 con 0x7f2404004d90 2023-10-14T11:59:57.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.652+0000 7f241959a700 1 -- 172.21.15.78:0/2119114322 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f240400ba10 con 0x7f241c0a70c0 2023-10-14T11:59:57.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.652+0000 7f241ad9d700 1 --2- 172.21.15.78:0/2119114322 >> v2:172.21.15.195:3300/0 conn(0x7f241c154ea0 0x7f241c15d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T11:59:57.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.652+0000 7f241a59c700 1 --2- 172.21.15.78:0/2119114322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2404003a00 0x7f2404003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:57.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.652+0000 7f2419d9b700 1 --2- 172.21.15.78:0/2119114322 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f241c0a70c0 0x7f2404004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:57.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.652+0000 7f241ad9d700 1 --2- 172.21.15.78:0/2119114322 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2404004d90 0x7f2404005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:57.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.652+0000 7f241a59c700 1 -- 172.21.15.78:0/2119114322 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f241c0a70c0 msgr2=0x7f2404004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:57.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.652+0000 7f241a59c700 1 --2- 172.21.15.78:0/2119114322 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f241c0a70c0 0x7f2404004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:57.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.652+0000 7f241a59c700 1 -- 172.21.15.78:0/2119114322 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2404004d90 msgr2=0x7f2404005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:57.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.652+0000 7f241a59c700 1 --2- 172.21.15.78:0/2119114322 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2404004d90 0x7f2404005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:57.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.652+0000 7f241a59c700 1 -- 172.21.15.78:0/2119114322 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f240c0094a0 con 0x7f2404003a00 2023-10-14T11:59:57.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.652+0000 7f241ad9d700 1 --2- 172.21.15.78:0/2119114322 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2404004d90 0x7f2404005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:59:57.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.652+0000 7f241a59c700 1 --2- 172.21.15.78:0/2119114322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2404003a00 0x7f2404003e20 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f241000c2f0 tx=0x7f2404005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea774fc054501023 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:57.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.653+0000 7f241959a700 1 -- 172.21.15.78:0/2119114322 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f241000ce30 con 0x7f2404003a00 2023-10-14T11:59:57.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.653+0000 7f241959a700 1 -- 172.21.15.78:0/2119114322 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2410005960 con 0x7f2404003a00 2023-10-14T11:59:57.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.653+0000 7f241959a700 1 -- 172.21.15.78:0/2119114322 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2410005d10 con 0x7f2404003a00 2023-10-14T11:59:57.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.653+0000 7f2420f58700 1 -- 172.21.15.78:0/2119114322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2404003a00 msgr2=0x7f2404003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:57.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.653+0000 7f2420f58700 1 --2- 172.21.15.78:0/2119114322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2404003a00 0x7f2404003e20 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f241000c2f0 tx=0x7f2404005880 comp rx=0 tx=0).stop 2023-10-14T11:59:57.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.653+0000 7f241ad9d700 1 -- 172.21.15.78:0/2119114322 reap_dead start 2023-10-14T11:59:57.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.653+0000 7f2420f58700 1 -- 172.21.15.78:0/2119114322 shutdown_connections 2023-10-14T11:59:57.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.653+0000 7f2420f58700 1 -- 172.21.15.78:0/2119114322 >> 172.21.15.78:0/2119114322 conn(0x7f241c0737d0 msgr2=0x7f241c073be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:57.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.653+0000 7f2420f58700 1 -- 172.21.15.78:0/2119114322 shutdown_connections 2023-10-14T11:59:57.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.653+0000 7f2420f58700 1 -- 172.21.15.78:0/2119114322 wait complete. 2023-10-14T11:59:57.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.654+0000 7f2420f58700 1 Processor -- start 2023-10-14T11:59:57.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.655+0000 7f2420f58700 1 -- start start 2023-10-14T11:59:57.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.655+0000 7f2420f58700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2404003a00 0x7f241c06d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:57.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.655+0000 7f2420f58700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f241c151e00 0x7f241c152280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:57.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.655+0000 7f2420f58700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f241c0c0660 0x7f241c0c0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:57.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.655+0000 7f2420f58700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241c0c7a70 con 0x7f241c151e00 2023-10-14T11:59:57.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.655+0000 7f2420f58700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241c0c4460 con 0x7f241c0c0660 2023-10-14T11:59:57.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.655+0000 7f2420f58700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f241c0bba00 con 0x7f2404003a00 2023-10-14T11:59:57.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.655+0000 7f2419d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f241c151e00 0x7f241c152280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:57.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.655+0000 7f2419d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f241c151e00 0x7f241c152280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47094/0 (socket says 172.21.15.78:47094) 2023-10-14T11:59:57.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.655+0000 7f2419d9b700 1 -- 172.21.15.78:0/1045537142 learned_addr learned my addr 172.21.15.78:0/1045537142 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:57.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.655+0000 7f241a59c700 1 --2- 172.21.15.78:0/1045537142 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2404003a00 0x7f241c06d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:57.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.655+0000 7f241ad9d700 1 --2- 172.21.15.78:0/1045537142 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f241c0c0660 0x7f241c0c0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:57.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.656+0000 7f2419d9b700 1 -- 172.21.15.78:0/1045537142 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2404003a00 msgr2=0x7f241c06d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:57.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.656+0000 7f2419d9b700 1 --2- 172.21.15.78:0/1045537142 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2404003a00 0x7f241c06d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:57.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.656+0000 7f2419d9b700 1 -- 172.21.15.78:0/1045537142 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f241c0c0660 msgr2=0x7f241c0c0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:57.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.656+0000 7f2419d9b700 1 --2- 172.21.15.78:0/1045537142 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f241c0c0660 0x7f241c0c0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:57.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.656+0000 7f2419d9b700 1 -- 172.21.15.78:0/1045537142 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f241c0c29e0 con 0x7f241c151e00 2023-10-14T11:59:57.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.656+0000 7f2419d9b700 1 --2- 172.21.15.78:0/1045537142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f241c151e00 0x7f241c152280 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f240c010a30 tx=0x7f240c010d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:57.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.656+0000 7f240affd700 1 -- 172.21.15.78:0/1045537142 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c010e70 con 0x7f241c151e00 2023-10-14T11:59:57.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.656+0000 7f240affd700 1 -- 172.21.15.78:0/1045537142 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f240c00e870 con 0x7f241c151e00 2023-10-14T11:59:57.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.656+0000 7f2420f58700 1 -- 172.21.15.78:0/1045537142 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f241c0c1b20 con 0x7f241c151e00 2023-10-14T11:59:57.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.656+0000 7f240affd700 1 -- 172.21.15.78:0/1045537142 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c00ebe0 con 0x7f241c151e00 2023-10-14T11:59:57.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.656+0000 7f2420f58700 1 -- 172.21.15.78:0/1045537142 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f241c1ea570 con 0x7f241c151e00 2023-10-14T11:59:57.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.657+0000 7f2420f58700 1 -- 172.21.15.78:0/1045537142 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f241c074b80 con 0x7f241c151e00 2023-10-14T11:59:57.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.657+0000 7f240affd700 1 -- 172.21.15.78:0/1045537142 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f240c00ed40 con 0x7f241c151e00 2023-10-14T11:59:57.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.658+0000 7f240affd700 1 --2- 172.21.15.78:0/1045537142 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f23fc06af00 0x7f23fc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:57.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.658+0000 7f241a59c700 1 --2- 172.21.15.78:0/1045537142 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f23fc06af00 0x7f23fc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:57.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.658+0000 7f240affd700 1 -- 172.21.15.78:0/1045537142 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f240c0895f0 con 0x7f241c151e00 2023-10-14T11:59:57.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.658+0000 7f241a59c700 1 --2- 172.21.15.78:0/1045537142 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f23fc06af00 0x7f23fc06d3c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2410008fd0 tx=0x7f241000a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:57.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.662+0000 7f240affd700 1 -- 172.21.15.78:0/1045537142 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f240c054a30 con 0x7f241c151e00 2023-10-14T11:59:57.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.852+0000 7f2420f58700 1 -- 172.21.15.78:0/1045537142 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"} v 0) v1 -- 0x7f241c0c0280 con 0x7f241c151e00 2023-10-14T11:59:57.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.857+0000 7f240affd700 1 -- 172.21.15.78:0/1045537142 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"}]=0 v13) v1 ==== 96+0+0 (secure 0 0 0) 0x7f240c009300 con 0x7f241c151e00 2023-10-14T11:59:57.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.860+0000 7f2420f58700 1 -- 172.21.15.78:0/1045537142 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f23fc06af00 msgr2=0x7f23fc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:57.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.860+0000 7f2420f58700 1 --2- 172.21.15.78:0/1045537142 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f23fc06af00 0x7f23fc06d3c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2410008fd0 tx=0x7f241000a000 comp rx=0 tx=0).stop 2023-10-14T11:59:57.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.860+0000 7f2420f58700 1 -- 172.21.15.78:0/1045537142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f241c151e00 msgr2=0x7f241c152280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:57.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.860+0000 7f2420f58700 1 --2- 172.21.15.78:0/1045537142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f241c151e00 0x7f241c152280 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f240c010a30 tx=0x7f240c010d40 comp rx=0 tx=0).stop 2023-10-14T11:59:57.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.860+0000 7f2420f58700 1 -- 172.21.15.78:0/1045537142 shutdown_connections 2023-10-14T11:59:57.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.860+0000 7f2420f58700 1 --2- 172.21.15.78:0/1045537142 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f23fc06af00 0x7f23fc06d3c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:57.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.860+0000 7f2420f58700 1 --2- 172.21.15.78:0/1045537142 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f241c0c0660 0x7f241c0c0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:57.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.860+0000 7f2420f58700 1 --2- 172.21.15.78:0/1045537142 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2404003a00 0x7f241c06d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:57.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.860+0000 7f2420f58700 1 --2- 172.21.15.78:0/1045537142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f241c151e00 0x7f241c152280 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:57.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.860+0000 7f2420f58700 1 -- 172.21.15.78:0/1045537142 >> 172.21.15.78:0/1045537142 conn(0x7f241c0737d0 msgr2=0x7f241c0a8040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:57.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.860+0000 7f2420f58700 1 -- 172.21.15.78:0/1045537142 shutdown_connections 2023-10-14T11:59:57.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.860+0000 7f2420f58700 1 -- 172.21.15.78:0/1045537142 wait complete. 2023-10-14T11:59:57.872 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-14T11:59:57.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.993+0000 7f54b0373700 1 Processor -- start 2023-10-14T11:59:57.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.993+0000 7f54b0373700 1 -- start start 2023-10-14T11:59:57.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.994+0000 7f54b0373700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f54a813dbe0 0x7f54a806a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:57.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.994+0000 7f54b0373700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f54a80c4540 0x7f54a80c02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:57.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.994+0000 7f54b0373700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a806ebe0 con 0x7f54a806ae10 2023-10-14T11:59:57.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.994+0000 7f54b0373700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54a806d330 con 0x7f54a80c4540 2023-10-14T11:59:57.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.994+0000 7f54b0373700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54a81558c0 con 0x7f54a813dbe0 2023-10-14T11:59:57.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.994+0000 7f54ad90e700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f54a80c4540 0x7f54a80c02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:57.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.995+0000 7f54ad90e700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f54a80c4540 0x7f54a80c02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47104/0 (socket says 172.21.15.78:47104) 2023-10-14T11:59:57.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.995+0000 7f54ad90e700 1 -- 172.21.15.78:0/1108624630 learned_addr learned my addr 172.21.15.78:0/1108624630 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:57.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.995+0000 7f54ad90e700 1 -- 172.21.15.78:0/1108624630 >> v2:172.21.15.195:3300/0 conn(0x7f54a813dbe0 msgr2=0x7f54a806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:57.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.995+0000 7f54ad90e700 1 --2- 172.21.15.78:0/1108624630 >> v2:172.21.15.195:3300/0 conn(0x7f54a813dbe0 0x7f54a806a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:57.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.995+0000 7f54ad90e700 1 -- 172.21.15.78:0/1108624630 >> v1:172.21.15.195:6789/0 conn(0x7f54a806ae10 legacy=0x7f54a80c0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:57.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.995+0000 7f54ad90e700 1 -- 172.21.15.78:0/1108624630 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a8154b00 con 0x7f54a80c4540 2023-10-14T11:59:57.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.995+0000 7f54ad90e700 1 --2- 172.21.15.78:0/1108624630 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54a80c4540 0x7f54a80c02e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f54a8072b60 tx=0x7f54a4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d993fa5d9dea796 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:57.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.995+0000 7f54ad10d700 1 -- 172.21.15.78:0/1108624630 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a4004230 con 0x7f54a80c4540 2023-10-14T11:59:58.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.995+0000 7f54ad10d700 1 -- 172.21.15.78:0/1108624630 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54a4004390 con 0x7f54a80c4540 2023-10-14T11:59:58.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.995+0000 7f54ad10d700 1 -- 172.21.15.78:0/1108624630 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a40036a0 con 0x7f54a80c4540 2023-10-14T11:59:58.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.996+0000 7f54b0373700 1 -- 172.21.15.78:0/1108624630 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54a80c4540 msgr2=0x7f54a80c02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:58.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.996+0000 7f54b0373700 1 --2- 172.21.15.78:0/1108624630 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54a80c4540 0x7f54a80c02e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f54a8072b60 tx=0x7f54a4009bd0 comp rx=0 tx=0).stop 2023-10-14T11:59:58.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.996+0000 7f54b0373700 1 -- 172.21.15.78:0/1108624630 shutdown_connections 2023-10-14T11:59:58.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.996+0000 7f54b0373700 1 --2- 172.21.15.78:0/1108624630 >> v2:172.21.15.195:3300/0 conn(0x7f54a813dbe0 0x7f54a806a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:58.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.996+0000 7f54b0373700 1 --2- 172.21.15.78:0/1108624630 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54a80c4540 0x7f54a80c02e0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:58.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.996+0000 7f54b0373700 1 -- 172.21.15.78:0/1108624630 >> 172.21.15.78:0/1108624630 conn(0x7f54a80c3a50 msgr2=0x7f54a80c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:58.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.996+0000 7f54b0373700 1 -- 172.21.15.78:0/1108624630 shutdown_connections 2023-10-14T11:59:58.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.996+0000 7f54b0373700 1 -- 172.21.15.78:0/1108624630 wait complete. 2023-10-14T11:59:58.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.997+0000 7f54b0373700 1 Processor -- start 2023-10-14T11:59:58.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.997+0000 7f54b0373700 1 -- start start 2023-10-14T11:59:58.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.998+0000 7f54b0373700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54a806ae10 0x7f54a813b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:58.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.998+0000 7f54b0373700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f54a80c4540 0x7f54a8156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:58.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.998+0000 7f54b0373700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54a813dbe0 0x7f54a8157430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:58.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.998+0000 7f54b0373700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a81ea5b0 con 0x7f54a813dbe0 2023-10-14T11:59:58.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.998+0000 7f54b0373700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a81eaae0 con 0x7f54a80c4540 2023-10-14T11:59:58.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.998+0000 7f54b0373700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54a8137b70 con 0x7f54a806ae10 2023-10-14T11:59:58.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.998+0000 7f54ae910700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54a813dbe0 0x7f54a8157430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:58.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.998+0000 7f54ae910700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54a813dbe0 0x7f54a8157430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47110/0 (socket says 172.21.15.78:47110) 2023-10-14T11:59:58.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.998+0000 7f54ae910700 1 -- 172.21.15.78:0/500984011 learned_addr learned my addr 172.21.15.78:0/500984011 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:58.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.998+0000 7f54ad90e700 1 --2- 172.21.15.78:0/500984011 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f54a80c4540 0x7f54a8156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:58.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.998+0000 7f54ae10f700 1 --2- 172.21.15.78:0/500984011 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54a806ae10 0x7f54a813b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:58.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.998+0000 7f54ae910700 1 -- 172.21.15.78:0/500984011 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54a806ae10 msgr2=0x7f54a813b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:58.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.998+0000 7f54ae910700 1 --2- 172.21.15.78:0/500984011 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54a806ae10 0x7f54a813b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:58.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.998+0000 7f54ae910700 1 -- 172.21.15.78:0/500984011 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f54a80c4540 msgr2=0x7f54a8156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:58.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.998+0000 7f54ae910700 1 --2- 172.21.15.78:0/500984011 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f54a80c4540 0x7f54a8156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:58.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.998+0000 7f54ae910700 1 -- 172.21.15.78:0/500984011 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a4009550 con 0x7f54a813dbe0 2023-10-14T11:59:58.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.998+0000 7f54ad90e700 1 --2- 172.21.15.78:0/500984011 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f54a80c4540 0x7f54a8156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:59:58.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.998+0000 7f54ae10f700 1 --2- 172.21.15.78:0/500984011 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54a806ae10 0x7f54a813b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T11:59:58.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.998+0000 7f54ae910700 1 --2- 172.21.15.78:0/500984011 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54a813dbe0 0x7f54a8157430 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f54a000c700 tx=0x7f54a000cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:58.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.998+0000 7f549effd700 1 -- 172.21.15.78:0/500984011 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a0003e10 con 0x7f54a813dbe0 2023-10-14T11:59:58.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.998+0000 7f549effd700 1 -- 172.21.15.78:0/500984011 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54a0005070 con 0x7f54a813dbe0 2023-10-14T11:59:58.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.998+0000 7f54b0373700 1 -- 172.21.15.78:0/500984011 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54a81ea800 con 0x7f54a813dbe0 2023-10-14T11:59:58.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.998+0000 7f549effd700 1 -- 172.21.15.78:0/500984011 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a0005420 con 0x7f54a813dbe0 2023-10-14T11:59:58.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.998+0000 7f54b0373700 1 -- 172.21.15.78:0/500984011 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54a8137800 con 0x7f54a813dbe0 2023-10-14T11:59:58.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:57.999+0000 7f54b0373700 1 -- 172.21.15.78:0/500984011 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54a8074b80 con 0x7f54a813dbe0 2023-10-14T11:59:58.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.000+0000 7f549effd700 1 -- 172.21.15.78:0/500984011 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f54a0005580 con 0x7f54a813dbe0 2023-10-14T11:59:58.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.000+0000 7f549effd700 1 --2- 172.21.15.78:0/500984011 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f548c06af00 0x7f548c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:58.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.000+0000 7f549effd700 1 -- 172.21.15.78:0/500984011 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f54a0088880 con 0x7f54a813dbe0 2023-10-14T11:59:58.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.000+0000 7f54ae10f700 1 --2- 172.21.15.78:0/500984011 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f548c06af00 0x7f548c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:58.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.001+0000 7f54ae10f700 1 --2- 172.21.15.78:0/500984011 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f548c06af00 0x7f548c06d3c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5498006fd0 tx=0x7f5498008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:58.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.005+0000 7f549effd700 1 -- 172.21.15.78:0/500984011 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f54a0053cc0 con 0x7f54a813dbe0 2023-10-14T11:59:58.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.198+0000 7f54b0373700 1 -- 172.21.15.78:0/500984011 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f54a806b2e0 con 0x7f54a813dbe0 2023-10-14T11:59:58.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.203+0000 7f549effd700 1 -- 172.21.15.78:0/500984011 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v14) v1 ==== 104+0+0 (secure 0 0 0) 0x7f54a0017020 con 0x7f54a813dbe0 2023-10-14T11:59:58.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.205+0000 7f54b0373700 1 -- 172.21.15.78:0/500984011 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f548c06af00 msgr2=0x7f548c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:58.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.205+0000 7f54b0373700 1 --2- 172.21.15.78:0/500984011 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f548c06af00 0x7f548c06d3c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5498006fd0 tx=0x7f5498008040 comp rx=0 tx=0).stop 2023-10-14T11:59:58.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.205+0000 7f54b0373700 1 -- 172.21.15.78:0/500984011 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54a813dbe0 msgr2=0x7f54a8157430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:58.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.205+0000 7f54b0373700 1 --2- 172.21.15.78:0/500984011 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54a813dbe0 0x7f54a8157430 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f54a000c700 tx=0x7f54a000cac0 comp rx=0 tx=0).stop 2023-10-14T11:59:58.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.205+0000 7f54b0373700 1 -- 172.21.15.78:0/500984011 shutdown_connections 2023-10-14T11:59:58.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.205+0000 7f54b0373700 1 --2- 172.21.15.78:0/500984011 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54a813dbe0 0x7f54a8157430 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:58.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.205+0000 7f54b0373700 1 --2- 172.21.15.78:0/500984011 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54a806ae10 0x7f54a813b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:58.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.205+0000 7f54b0373700 1 --2- 172.21.15.78:0/500984011 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f548c06af00 0x7f548c06d3c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:58.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.205+0000 7f54b0373700 1 --2- 172.21.15.78:0/500984011 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f54a80c4540 0x7f54a8156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:58.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.205+0000 7f54b0373700 1 -- 172.21.15.78:0/500984011 >> 172.21.15.78:0/500984011 conn(0x7f54a80c3a50 msgr2=0x7f54a80c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:58.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.205+0000 7f54b0373700 1 -- 172.21.15.78:0/500984011 shutdown_connections 2023-10-14T11:59:58.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.205+0000 7f54b0373700 1 -- 172.21.15.78:0/500984011 wait complete. 2023-10-14T11:59:58.218 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop' 2023-10-14T11:59:58.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.338+0000 7f19b304d700 1 Processor -- start 2023-10-14T11:59:58.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.338+0000 7f19b304d700 1 -- start start 2023-10-14T11:59:58.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.339+0000 7f19b304d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f19ac0c44c0 0x7f19ac0c7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:58.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.339+0000 7f19b304d700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f19ac0c72a0 0x7f19ac0c86e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:58.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.339+0000 7f19b304d700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f19ac0c8c20 0x7f19ac0c2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:58.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.339+0000 7f19b304d700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19ac156b20 con 0x7f19ac0c72a0 2023-10-14T11:59:58.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.339+0000 7f19b304d700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19ac155f10 con 0x7f19ac0c44c0 2023-10-14T11:59:58.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.339+0000 7f19b304d700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19ac1561f0 con 0x7f19ac0c8c20 2023-10-14T11:59:58.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.339+0000 7f19abfff700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f19ac0c72a0 0x7f19ac0c86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:58.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.339+0000 7f19abfff700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f19ac0c72a0 0x7f19ac0c86e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47114/0 (socket says 172.21.15.78:47114) 2023-10-14T11:59:58.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.339+0000 7f19abfff700 1 -- 172.21.15.78:0/2241990706 learned_addr learned my addr 172.21.15.78:0/2241990706 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:58.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.340+0000 7f19abfff700 1 -- 172.21.15.78:0/2241990706 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f19ac0c8c20 msgr2=0x7f19ac0c2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:58.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.340+0000 7f19abfff700 1 --2- 172.21.15.78:0/2241990706 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f19ac0c8c20 0x7f19ac0c2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:58.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.340+0000 7f19abfff700 1 -- 172.21.15.78:0/2241990706 >> v2:172.21.15.195:3300/0 conn(0x7f19ac0c44c0 msgr2=0x7f19ac0c7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T11:59:58.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.340+0000 7f19b0de9700 1 --2- 172.21.15.78:0/2241990706 >> v2:172.21.15.195:3300/0 conn(0x7f19ac0c44c0 0x7f19ac0c7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:58.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.340+0000 7f19abfff700 1 --2- 172.21.15.78:0/2241990706 >> v2:172.21.15.195:3300/0 conn(0x7f19ac0c44c0 0x7f19ac0c7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:58.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.340+0000 7f19abfff700 1 -- 172.21.15.78:0/2241990706 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ac1564d0 con 0x7f19ac0c72a0 2023-10-14T11:59:58.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.340+0000 7f19abfff700 1 --2- 172.21.15.78:0/2241990706 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19ac0c72a0 0x7f19ac0c86e0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f19ac0c9210 tx=0x7f1998009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34f94932e08708a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:58.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.340+0000 7f19ab7fe700 1 -- 172.21.15.78:0/2241990706 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199800b870 con 0x7f19ac0c72a0 2023-10-14T11:59:58.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.340+0000 7f19ab7fe700 1 -- 172.21.15.78:0/2241990706 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f199800b9d0 con 0x7f19ac0c72a0 2023-10-14T11:59:58.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.340+0000 7f19ab7fe700 1 -- 172.21.15.78:0/2241990706 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199800bd60 con 0x7f19ac0c72a0 2023-10-14T11:59:58.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.341+0000 7f19b304d700 1 -- 172.21.15.78:0/2241990706 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19ac0c72a0 msgr2=0x7f19ac0c86e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:58.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.341+0000 7f19b304d700 1 --2- 172.21.15.78:0/2241990706 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19ac0c72a0 0x7f19ac0c86e0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f19ac0c9210 tx=0x7f1998009bd0 comp rx=0 tx=0).stop 2023-10-14T11:59:58.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.341+0000 7f19b304d700 1 -- 172.21.15.78:0/2241990706 shutdown_connections 2023-10-14T11:59:58.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.341+0000 7f19b304d700 1 --2- 172.21.15.78:0/2241990706 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f19ac0c8c20 0x7f19ac0c2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:58.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.341+0000 7f19b304d700 1 --2- 172.21.15.78:0/2241990706 >> v2:172.21.15.195:3300/0 conn(0x7f19ac0c44c0 0x7f19ac0c7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:58.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.341+0000 7f19b304d700 1 --2- 172.21.15.78:0/2241990706 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19ac0c72a0 0x7f19ac0c86e0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:58.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.341+0000 7f19b304d700 1 -- 172.21.15.78:0/2241990706 >> 172.21.15.78:0/2241990706 conn(0x7f19ac0737d0 msgr2=0x7f19ac073be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:58.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.341+0000 7f19b304d700 1 -- 172.21.15.78:0/2241990706 shutdown_connections 2023-10-14T11:59:58.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.341+0000 7f19b304d700 1 -- 172.21.15.78:0/2241990706 wait complete. 2023-10-14T11:59:58.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.342+0000 7f19b304d700 1 Processor -- start 2023-10-14T11:59:58.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.342+0000 7f19b304d700 1 -- start start 2023-10-14T11:59:58.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.343+0000 7f19b304d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19ac0c44c0 0x7f19ac0bbc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:58.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.343+0000 7f19b304d700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f19ac0c72a0 0x7f19ac0c1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:58.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.343+0000 7f19b304d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f19ac0c8c20 0x7f19ac152050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:58.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.343+0000 7f19b304d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ac1e79e0 con 0x7f19ac0c44c0 2023-10-14T11:59:58.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.343+0000 7f19b304d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ac0bf7b0 con 0x7f19ac0c8c20 2023-10-14T11:59:58.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.343+0000 7f19b304d700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19ac0bd550 con 0x7f19ac0c72a0 2023-10-14T11:59:58.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.343+0000 7f19b0de9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19ac0c44c0 0x7f19ac0bbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:58.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.343+0000 7f19b0de9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19ac0c44c0 0x7f19ac0bbc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47120/0 (socket says 172.21.15.78:47120) 2023-10-14T11:59:58.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.343+0000 7f19b0de9700 1 -- 172.21.15.78:0/1887507723 learned_addr learned my addr 172.21.15.78:0/1887507723 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T11:59:58.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.343+0000 7f19b15ea700 1 --2- 172.21.15.78:0/1887507723 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f19ac0c8c20 0x7f19ac152050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:58.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.343+0000 7f19abfff700 1 --2- 172.21.15.78:0/1887507723 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f19ac0c72a0 0x7f19ac0c1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:58.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.344+0000 7f19b0de9700 1 -- 172.21.15.78:0/1887507723 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f19ac0c72a0 msgr2=0x7f19ac0c1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:58.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.344+0000 7f19b0de9700 1 --2- 172.21.15.78:0/1887507723 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f19ac0c72a0 0x7f19ac0c1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:58.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.344+0000 7f19b0de9700 1 -- 172.21.15.78:0/1887507723 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f19ac0c8c20 msgr2=0x7f19ac152050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:58.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.344+0000 7f19b0de9700 1 --2- 172.21.15.78:0/1887507723 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f19ac0c8c20 0x7f19ac152050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:58.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.344+0000 7f19b0de9700 1 -- 172.21.15.78:0/1887507723 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1998009550 con 0x7f19ac0c44c0 2023-10-14T11:59:58.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.344+0000 7f19b15ea700 1 --2- 172.21.15.78:0/1887507723 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f19ac0c8c20 0x7f19ac152050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:59:58.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.344+0000 7f19abfff700 1 --2- 172.21.15.78:0/1887507723 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f19ac0c72a0 0x7f19ac0c1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T11:59:58.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.344+0000 7f19b0de9700 1 --2- 172.21.15.78:0/1887507723 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19ac0c44c0 0x7f19ac0bbc90 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f19a000c7f0 tx=0x7f19a000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:58.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.344+0000 7f19a97fa700 1 -- 172.21.15.78:0/1887507723 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a0011800 con 0x7f19ac0c44c0 2023-10-14T11:59:58.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.344+0000 7f19a97fa700 1 -- 172.21.15.78:0/1887507723 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19a0011960 con 0x7f19ac0c44c0 2023-10-14T11:59:58.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.344+0000 7f19b304d700 1 -- 172.21.15.78:0/1887507723 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19ac1ebcd0 con 0x7f19ac0c44c0 2023-10-14T11:59:58.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.344+0000 7f19a97fa700 1 -- 172.21.15.78:0/1887507723 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a0011cf0 con 0x7f19ac0c44c0 2023-10-14T11:59:58.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.344+0000 7f19b304d700 1 -- 172.21.15.78:0/1887507723 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19ac0bfce0 con 0x7f19ac0c44c0 2023-10-14T11:59:58.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.345+0000 7f19b304d700 1 -- 172.21.15.78:0/1887507723 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19ac074b80 con 0x7f19ac0c44c0 2023-10-14T11:59:58.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.345+0000 7f19a97fa700 1 -- 172.21.15.78:0/1887507723 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f19a0018360 con 0x7f19ac0c44c0 2023-10-14T11:59:58.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.346+0000 7f19a97fa700 1 --2- 172.21.15.78:0/1887507723 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f199006af00 0x7f199006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T11:59:58.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.346+0000 7f19a97fa700 1 -- 172.21.15.78:0/1887507723 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f19a00892c0 con 0x7f19ac0c44c0 2023-10-14T11:59:58.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.346+0000 7f19abfff700 1 --2- 172.21.15.78:0/1887507723 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f199006af00 0x7f199006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T11:59:58.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.346+0000 7f19abfff700 1 --2- 172.21.15.78:0/1887507723 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f199006af00 0x7f199006d3c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f19ac0c3060 tx=0x7f1998008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T11:59:58.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.350+0000 7f19a97fa700 1 -- 172.21.15.78:0/1887507723 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f19a0054680 con 0x7f19ac0c44c0 2023-10-14T11:59:58.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:58.623+0000 7f19b304d700 1 -- 172.21.15.78:0/1887507723 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop"]} v 0) v1 -- 0x7f19ac0c7080 con 0x7f19ac0c44c0 2023-10-14T11:59:59.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:59.513+0000 7f19a97fa700 1 -- 172.21.15.78:0/1887507723 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop"]}]=0 v100) v1 ==== 143+0+0 (secure 0 0 0) 0x7f19a0057ca0 con 0x7f19ac0c44c0 2023-10-14T11:59:59.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:59.517+0000 7f19b304d700 1 -- 172.21.15.78:0/1887507723 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f199006af00 msgr2=0x7f199006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:59.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:59.517+0000 7f19b304d700 1 --2- 172.21.15.78:0/1887507723 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f199006af00 0x7f199006d3c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f19ac0c3060 tx=0x7f1998008030 comp rx=0 tx=0).stop 2023-10-14T11:59:59.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:59.517+0000 7f19b304d700 1 -- 172.21.15.78:0/1887507723 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19ac0c44c0 msgr2=0x7f19ac0bbc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T11:59:59.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:59.517+0000 7f19b304d700 1 --2- 172.21.15.78:0/1887507723 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19ac0c44c0 0x7f19ac0bbc90 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f19a000c7f0 tx=0x7f19a000cbb0 comp rx=0 tx=0).stop 2023-10-14T11:59:59.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:59.517+0000 7f19b304d700 1 -- 172.21.15.78:0/1887507723 shutdown_connections 2023-10-14T11:59:59.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:59.517+0000 7f19b304d700 1 --2- 172.21.15.78:0/1887507723 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f199006af00 0x7f199006d3c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:59.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:59.517+0000 7f19b304d700 1 --2- 172.21.15.78:0/1887507723 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f19ac0c8c20 0x7f19ac152050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:59.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:59.517+0000 7f19b304d700 1 --2- 172.21.15.78:0/1887507723 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19ac0c44c0 0x7f19ac0bbc90 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:59.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:59.517+0000 7f19b304d700 1 --2- 172.21.15.78:0/1887507723 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f19ac0c72a0 0x7f19ac0c1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T11:59:59.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:59.517+0000 7f19b304d700 1 -- 172.21.15.78:0/1887507723 >> 172.21.15.78:0/1887507723 conn(0x7f19ac0737d0 msgr2=0x7f19ac0c4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T11:59:59.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:59.517+0000 7f19b304d700 1 -- 172.21.15.78:0/1887507723 shutdown_connections 2023-10-14T11:59:59.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T11:59:59.517+0000 7f19b304d700 1 -- 172.21.15.78:0/1887507723 wait complete. 2023-10-14T11:59:59.533 INFO:tasks.cephfs_test_runner:That unsetting mds_join_fs will not cause failover if no high-affinity standby exists. ... ERROR 2023-10-14T11:59:59.533 INFO:tasks.cephfs_test_runner: 2023-10-14T11:59:59.534 INFO:tasks.cephfs_test_runner:====================================================================== 2023-10-14T11:59:59.534 INFO:tasks.cephfs_test_runner:ERROR: test_join_fs_drop (tasks.cephfs.test_failover.TestClusterAffinity) 2023-10-14T11:59:59.534 INFO:tasks.cephfs_test_runner:That unsetting mds_join_fs will not cause failover if no high-affinity standby exists. 2023-10-14T11:59:59.535 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-10-14T11:59:59.535 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-10-14T11:59:59.535 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/qa/tasks/cephfs/test_failover.py", line 104, in test_join_fs_drop 2023-10-14T11:59:59.535 INFO:tasks.cephfs_test_runner: self._reach_target(target) 2023-10-14T11:59:59.536 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/qa/tasks/cephfs/test_failover.py", line 61, in _reach_target 2023-10-14T11:59:59.536 INFO:tasks.cephfs_test_runner: self.wait_until_true(takeover, 30) 2023-10-14T11:59:59.536 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/qa/tasks/ceph_test_case.py", line 345, in wait_until_true 2023-10-14T11:59:59.536 INFO:tasks.cephfs_test_runner: raise TestTimeoutError("Timed out after {0}s and {1} retries".format(elapsed, retry_count)) 2023-10-14T11:59:59.537 INFO:tasks.cephfs_test_runner:tasks.ceph_test_case.TestTimeoutError: Timed out after 30s and 0 retries 2023-10-14T11:59:59.537 INFO:tasks.cephfs_test_runner: 2023-10-14T11:59:59.537 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-10-14T11:59:59.537 INFO:tasks.cephfs_test_runner:Ran 1 test in 79.107s 2023-10-14T11:59:59.538 INFO:tasks.cephfs_test_runner: 2023-10-14T11:59:59.538 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2023-10-14T11:59:59.538 INFO:tasks.cephfs_test_runner: 2023-10-14T11:59:59.538 INFO:tasks.cephfs_test_runner:====================================================================== 2023-10-14T11:59:59.539 INFO:tasks.cephfs_test_runner:ERROR: test_join_fs_drop (tasks.cephfs.test_failover.TestClusterAffinity) 2023-10-14T11:59:59.539 INFO:tasks.cephfs_test_runner:That unsetting mds_join_fs will not cause failover if no high-affinity standby exists. 2023-10-14T11:59:59.540 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-10-14T11:59:59.540 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-10-14T11:59:59.540 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/qa/tasks/cephfs/test_failover.py", line 104, in test_join_fs_drop 2023-10-14T11:59:59.540 INFO:tasks.cephfs_test_runner: self._reach_target(target) 2023-10-14T11:59:59.541 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/qa/tasks/cephfs/test_failover.py", line 61, in _reach_target 2023-10-14T11:59:59.541 INFO:tasks.cephfs_test_runner: self.wait_until_true(takeover, 30) 2023-10-14T11:59:59.541 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/qa/tasks/ceph_test_case.py", line 345, in wait_until_true 2023-10-14T11:59:59.541 INFO:tasks.cephfs_test_runner: raise TestTimeoutError("Timed out after {0}s and {1} retries".format(elapsed, retry_count)) 2023-10-14T11:59:59.542 INFO:tasks.cephfs_test_runner:tasks.ceph_test_case.TestTimeoutError: Timed out after 30s and 0 retries 2023-10-14T11:59:59.542 INFO:tasks.cephfs_test_runner: 2023-10-14T11:59:59.542 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_join_fs_drop (tasks.cephfs.test_failover.TestClusterAffinity) 2023-10-14T11:59:59.605 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=d85300d859db4d299817149550aafc40 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_join_fs_drop (tasks.cephfs.test_failover.TestClusterAffinity) 2023-10-14T11:59:59.607 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2023-10-14T11:59:59.616 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-10-14T11:59:59.635 INFO:tasks.kclient:Unmounting kernel clients... 2023-10-14T11:59:59.635 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-14T11:59:59.636 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T11:59:59.673 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17259 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 2023-10-14T11:59:59.673 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-14T11:59:59.673 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T11:59:59.696 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15735 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 2023-10-14T11:59:59.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:59:59.697 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2023-10-14T11:59:59.733 INFO:teuthology.orchestra.run.smithi078.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-14T11:59:59.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:59:59.734 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-14T11:59:59.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:59:59.813 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link delete ceph-brx 2023-10-14T11:59:59.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:59:59.925 DEBUG:teuthology.orchestra.run.smithi195:> ip netns list 2023-10-14T11:59:59.946 INFO:teuthology.orchestra.run.smithi195.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-10-14T11:59:59.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T11:59:59.947 DEBUG:teuthology.orchestra.run.smithi195:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-14T12:00:00.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T12:00:00.034 DEBUG:teuthology.orchestra.run.smithi195:> sudo ip link delete ceph-brx 2023-10-14T12:00:00.137 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-14T12:00:00.148 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-14T12:00:00.149 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T12:00:00.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.274+0000 7f8cf3315700 1 Processor -- start 2023-10-14T12:00:00.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.274+0000 7f8cf3315700 1 -- start start 2023-10-14T12:00:00.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.275+0000 7f8cf3315700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8cec14fd80 0x7f8cec156850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:00.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.275+0000 7f8cf3315700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8cec0c1840 0x7f8cec1531f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:00.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.275+0000 7f8cf3315700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cec155de0 con 0x7f8cec070240 2023-10-14T12:00:00.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.275+0000 7f8cf3315700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cec0a85d0 con 0x7f8cec0c1840 2023-10-14T12:00:00.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.275+0000 7f8cf3315700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec153730 con 0x7f8cec14fd80 2023-10-14T12:00:00.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.275+0000 7f8cf08b0700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8cec0c1840 0x7f8cec1531f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:00.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.275+0000 7f8cf08b0700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8cec0c1840 0x7f8cec1531f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47128/0 (socket says 172.21.15.78:47128) 2023-10-14T12:00:00.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.275+0000 7f8cf18b2700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f8cec070240 0x7f8cec071dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52008/0 (socket says 172.21.15.78:52008) 2023-10-14T12:00:00.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.275+0000 7f8cf08b0700 1 -- 172.21.15.78:0/1691955893 learned_addr learned my addr 172.21.15.78:0/1691955893 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:00.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.276+0000 7f8cf08b0700 1 -- 172.21.15.78:0/1691955893 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8cec14fd80 msgr2=0x7f8cec156850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T12:00:00.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.276+0000 7f8cf08b0700 1 --2- 172.21.15.78:0/1691955893 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8cec14fd80 0x7f8cec156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:00.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.276+0000 7f8cf08b0700 1 -- 172.21.15.78:0/1691955893 >> v1:172.21.15.78:6789/0 conn(0x7f8cec070240 legacy=0x7f8cec071dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:00.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.276+0000 7f8cf08b0700 1 -- 172.21.15.78:0/1691955893 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cec150160 con 0x7f8cec0c1840 2023-10-14T12:00:00.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.276+0000 7f8ce3fff700 1 -- 172.21.15.78:0/1691955893 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 15574742 0 0) 0x7f8cec155de0 con 0x7f8cec070240 2023-10-14T12:00:00.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.276+0000 7f8cf08b0700 1 --2- 172.21.15.78:0/1691955893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cec0c1840 0x7f8cec1531f0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f8ce80098f0 tx=0x7f8ce8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=392ac7996a84cf0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:00.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.276+0000 7f8ce3fff700 1 -- 172.21.15.78:0/1691955893 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce800b8a0 con 0x7f8cec0c1840 2023-10-14T12:00:00.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.276+0000 7f8ce3fff700 1 -- 172.21.15.78:0/1691955893 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ce800ba00 con 0x7f8cec0c1840 2023-10-14T12:00:00.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.276+0000 7f8ce3fff700 1 -- 172.21.15.78:0/1691955893 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce800bd90 con 0x7f8cec0c1840 2023-10-14T12:00:00.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.276+0000 7f8cf3315700 1 -- 172.21.15.78:0/1691955893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cec0c1840 msgr2=0x7f8cec1531f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:00.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.276+0000 7f8cf3315700 1 --2- 172.21.15.78:0/1691955893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cec0c1840 0x7f8cec1531f0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f8ce80098f0 tx=0x7f8ce8009c00 comp rx=0 tx=0).stop 2023-10-14T12:00:00.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.277+0000 7f8cf3315700 1 -- 172.21.15.78:0/1691955893 shutdown_connections 2023-10-14T12:00:00.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.277+0000 7f8cf3315700 1 --2- 172.21.15.78:0/1691955893 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8cec14fd80 0x7f8cec156850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:00.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.277+0000 7f8cf3315700 1 --2- 172.21.15.78:0/1691955893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cec0c1840 0x7f8cec1531f0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:00.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.277+0000 7f8cf3315700 1 -- 172.21.15.78:0/1691955893 >> 172.21.15.78:0/1691955893 conn(0x7f8cec0c2930 msgr2=0x7f8cec0c2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:00.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.277+0000 7f8cf3315700 1 -- 172.21.15.78:0/1691955893 shutdown_connections 2023-10-14T12:00:00.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.277+0000 7f8cf3315700 1 -- 172.21.15.78:0/1691955893 wait complete. 2023-10-14T12:00:00.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.278+0000 7f8cf3315700 1 Processor -- start 2023-10-14T12:00:00.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.278+0000 7f8cf3315700 1 -- start start 2023-10-14T12:00:00.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.278+0000 7f8cf3315700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8cec070240 0x7f8cec1eb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:00.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.278+0000 7f8cf3315700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8cec0c1840 0x7f8cec149c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:00.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.278+0000 7f8cf3315700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cec14fd80 0x7f8cec0beff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:00.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.278+0000 7f8cf3315700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec1505a0 con 0x7f8cec14fd80 2023-10-14T12:00:00.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.278+0000 7f8cf3315700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec0c03a0 con 0x7f8cec0c1840 2023-10-14T12:00:00.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.278+0000 7f8cf3315700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec154e50 con 0x7f8cec070240 2023-10-14T12:00:00.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.279+0000 7f8cf18b2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cec14fd80 0x7f8cec0beff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:00.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.279+0000 7f8cf18b2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cec14fd80 0x7f8cec0beff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47138/0 (socket says 172.21.15.78:47138) 2023-10-14T12:00:00.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.279+0000 7f8cf18b2700 1 -- 172.21.15.78:0/662951485 learned_addr learned my addr 172.21.15.78:0/662951485 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:00.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.279+0000 7f8cf08b0700 1 --2- 172.21.15.78:0/662951485 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8cec0c1840 0x7f8cec149c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:00.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.279+0000 7f8cf10b1700 1 --2- 172.21.15.78:0/662951485 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8cec070240 0x7f8cec1eb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:00.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.279+0000 7f8cf18b2700 1 -- 172.21.15.78:0/662951485 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8cec070240 msgr2=0x7f8cec1eb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:00.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.279+0000 7f8cf18b2700 1 --2- 172.21.15.78:0/662951485 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8cec070240 0x7f8cec1eb8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:00.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.279+0000 7f8cf18b2700 1 -- 172.21.15.78:0/662951485 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8cec0c1840 msgr2=0x7f8cec149c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:00.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.279+0000 7f8cf18b2700 1 --2- 172.21.15.78:0/662951485 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8cec0c1840 0x7f8cec149c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:00.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.279+0000 7f8cf18b2700 1 -- 172.21.15.78:0/662951485 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce8009550 con 0x7f8cec14fd80 2023-10-14T12:00:00.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.279+0000 7f8cf18b2700 1 --2- 172.21.15.78:0/662951485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cec14fd80 0x7f8cec0beff0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f8ce4002230 tx=0x7f8ce400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:00.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.279+0000 7f8ce1ffb700 1 -- 172.21.15.78:0/662951485 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce400ac20 con 0x7f8cec14fd80 2023-10-14T12:00:00.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.279+0000 7f8ce1ffb700 1 -- 172.21.15.78:0/662951485 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ce400ada0 con 0x7f8cec14fd80 2023-10-14T12:00:00.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.279+0000 7f8ce1ffb700 1 -- 172.21.15.78:0/662951485 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce4007310 con 0x7f8cec14fd80 2023-10-14T12:00:00.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.279+0000 7f8cf3315700 1 -- 172.21.15.78:0/662951485 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cec155130 con 0x7f8cec14fd80 2023-10-14T12:00:00.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.279+0000 7f8cf3315700 1 -- 172.21.15.78:0/662951485 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cec14a1a0 con 0x7f8cec14fd80 2023-10-14T12:00:00.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.280+0000 7f8cf3315700 1 -- 172.21.15.78:0/662951485 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cec150710 con 0x7f8cec14fd80 2023-10-14T12:00:00.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.281+0000 7f8ce1ffb700 1 -- 172.21.15.78:0/662951485 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8ce4007470 con 0x7f8cec14fd80 2023-10-14T12:00:00.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.281+0000 7f8ce1ffb700 1 --2- 172.21.15.78:0/662951485 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8cd006af00 0x7f8cd006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:00.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.281+0000 7f8ce1ffb700 1 -- 172.21.15.78:0/662951485 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8ce4088e10 con 0x7f8cec14fd80 2023-10-14T12:00:00.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.281+0000 7f8cf10b1700 1 --2- 172.21.15.78:0/662951485 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8cd006af00 0x7f8cd006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:00.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.281+0000 7f8cf10b1700 1 --2- 172.21.15.78:0/662951485 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8cd006af00 0x7f8cd006d3c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8cdc006fd0 tx=0x7f8cdc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:00.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.285+0000 7f8ce1ffb700 1 -- 172.21.15.78:0/662951485 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8ce40541c0 con 0x7f8cec14fd80 2023-10-14T12:00:00.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.470+0000 7f8cf3315700 1 -- 172.21.15.78:0/662951485 --> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8cec156050 con 0x7f8cd006af00 2023-10-14T12:00:00.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.473+0000 7f8ce1ffb700 1 -- 172.21.15.78:0/662951485 <== mgr.4104 v2:172.21.15.78:6832/3171890074 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+51053 (secure 0 0 0) 0x7f8cdc0123e0 con 0x7f8cd006af00 2023-10-14T12:00:00.474 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T12:00:00.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.475+0000 7f8cf3315700 1 -- 172.21.15.78:0/662951485 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8cd006af00 msgr2=0x7f8cd006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:00.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.475+0000 7f8cf3315700 1 --2- 172.21.15.78:0/662951485 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8cd006af00 0x7f8cd006d3c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8cdc006fd0 tx=0x7f8cdc008040 comp rx=0 tx=0).stop 2023-10-14T12:00:00.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.475+0000 7f8cf3315700 1 -- 172.21.15.78:0/662951485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cec14fd80 msgr2=0x7f8cec0beff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:00.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.475+0000 7f8cf3315700 1 --2- 172.21.15.78:0/662951485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cec14fd80 0x7f8cec0beff0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f8ce4002230 tx=0x7f8ce400a060 comp rx=0 tx=0).stop 2023-10-14T12:00:00.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.475+0000 7f8cf3315700 1 -- 172.21.15.78:0/662951485 shutdown_connections 2023-10-14T12:00:00.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.475+0000 7f8cf3315700 1 --2- 172.21.15.78:0/662951485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cec14fd80 0x7f8cec0beff0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:00.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.475+0000 7f8cf3315700 1 --2- 172.21.15.78:0/662951485 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8cec070240 0x7f8cec1eb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:00.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.475+0000 7f8cf3315700 1 --2- 172.21.15.78:0/662951485 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f8cd006af00 0x7f8cd006d3c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:00.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.475+0000 7f8cf3315700 1 --2- 172.21.15.78:0/662951485 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8cec0c1840 0x7f8cec149c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:00.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.475+0000 7f8cf3315700 1 -- 172.21.15.78:0/662951485 >> 172.21.15.78:0/662951485 conn(0x7f8cec0c2930 msgr2=0x7f8cec0a8720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:00.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.475+0000 7f8cf3315700 1 -- 172.21.15.78:0/662951485 shutdown_connections 2023-10-14T12:00:00.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.475+0000 7f8cf3315700 1 -- 172.21.15.78:0/662951485 wait complete. 2023-10-14T12:00:00.480 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2023-10-14T12:00:00.487 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2023-10-14T11:59:58.989175+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219296,"kb_used_data":3124,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755424,"statfs":{"total":773094113280,"available":772869554176,"internally_reserved":0,"allocated":3198976,"data_stored":2361333,"data_compressed":65244,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2410,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-45,"num_write_kb":-14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001208"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":175,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T11:59:55.540323+0000","last_change":"2023-10-14T11:58:10.616726+0000","last_active":"2023-10-14T11:59:55.540323+0000","last_peered":"2023-10-14T11:59:55.540323+0000","last_clean":"2023-10-14T11:59:55.540323+0000","last_became_active":"2023-10-14T11:58:10.615628+0000","last_became_peered":"2023-10-14T11:58:10.615628+0000","last_unstale":"2023-10-14T11:59:55.540323+0000","last_undegraded":"2023-10-14T11:59:55.540323+0000","last_fullsized":"2023-10-14T11:59:55.540323+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:09.001265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:09.001265+0000","last_clean_scrub_stamp":"2023-10-14T11:58:09.001265+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:22:50.496050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65244,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574873,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":454656,"data_stored":741333,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.441}]},{"osd":1,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.317},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.35899999999999999}]},{"osd":2,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":2.8250000000000002,"5min":2.8250000000000002,"15min":2.8250000000000002},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.46300000000000002}]},{"osd":3,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":2.8639999999999999,"5min":2.8639999999999999,"15min":2.8639999999999999},"last":0.66800000000000004}]},{"osd":4,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.47699999999999998}]},{"osd":5,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.498},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.57099999999999995}]},{"osd":6,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.54800000000000004}]}]},{"osd":6,"up_from":13,"seq":55834574873,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":145558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.57099999999999995}]},{"osd":1,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.61399999999999999}]},{"osd":2,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.54600000000000004}]},{"osd":3,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":2.2789999999999999,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.496}]},{"osd":4,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.43099999999999999}]},{"osd":5,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.47999999999999998}]},{"osd":7,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.65700000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574873,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":146576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.30599999999999999}]},{"osd":2,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.37}]},{"osd":3,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.495,"5min":0.495,"15min":0.495},"last":0.34599999999999997}]},{"osd":4,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.72199999999999998}]},{"osd":5,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.57399999999999995}]},{"osd":6,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.68300000000000005}]},{"osd":7,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.58799999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574872,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27448,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344392,"statfs":{"total":96636764160,"available":96608657408,"internally_reserved":0,"allocated":438272,"data_stored":735310,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.31,"15min":0.31},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"last":0.26500000000000001}]},{"osd":2,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.378,"5min":0.378,"15min":0.378},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.318}]},{"osd":3,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.439,"5min":0.439,"15min":0.439},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.41799999999999998}]},{"osd":4,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.53400000000000003}]},{"osd":5,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.55600000000000005}]},{"osd":6,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.52100000000000002}]},{"osd":7,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.47099999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574872,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":376832,"data_stored":145012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 11:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.28799999999999998}]},{"osd":1,"last update":"Sat Oct 14 11:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.37},{"interface":"front","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.35599999999999998}]},{"osd":3,"last update":"Sat Oct 14 11:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.33100000000000002}]},{"osd":4,"last update":"Sat Oct 14 11:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1,"5min":1,"15min":1},"last":0.437},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.57999999999999996}]},{"osd":5,"last update":"Sat Oct 14 11:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.48199999999999998}]},{"osd":6,"last update":"Sat Oct 14 11:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.42399999999999999}]},{"osd":7,"last update":"Sat Oct 14 11:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.622},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.44600000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":150965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 11:59:10 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.226,"5min":7.226,"15min":7.226},"last":0.25600000000000001},{"interface":"front","average":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":21.067,"5min":21.067,"15min":21.067},"last":0.30499999999999999}]},{"osd":1,"last update":"Sat Oct 14 11:59:10 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":7.3419999999999996,"5min":7.3419999999999996,"15min":7.3419999999999996},"last":0.36599999999999999}]},{"osd":2,"last update":"Sat Oct 14 11:59:10 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":7.3120000000000003,"5min":7.3120000000000003,"15min":7.3120000000000003},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":7.1870000000000003,"5min":7.1870000000000003,"15min":7.1870000000000003},"last":0.32800000000000001}]},{"osd":4,"last update":"Sat Oct 14 11:59:10 2023","interfaces":[{"interface":"back","average":{"1min":1.8640000000000001,"5min":1.8640000000000001,"15min":1.8640000000000001},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":20.984000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":17.902999999999999,"5min":17.902999999999999,"15min":17.902999999999999},"last":0.56799999999999995}]},{"osd":5,"last update":"Sat Oct 14 11:59:10 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":7.2560000000000002,"5min":7.2560000000000002,"15min":7.2560000000000002},"last":0.55100000000000005},{"interface":"front","average":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":21.100000000000001,"5min":21.100000000000001,"15min":21.100000000000001},"last":0.46500000000000002}]},{"osd":6,"last update":"Sat Oct 14 11:59:10 2023","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.002,"15min":1.002},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":7.1589999999999998,"5min":7.1589999999999998,"15min":7.1589999999999998},"last":0.47399999999999998},{"interface":"front","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"last":0.58999999999999997}]},{"osd":7,"last update":"Sat Oct 14 11:59:10 2023","interfaces":[{"interface":"back","average":{"1min":1.891,"5min":1.891,"15min":1.891},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":21.035,"5min":21.035,"15min":21.035},"last":0.48599999999999999},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":7.1230000000000002,"5min":7.1230000000000002,"15min":7.1230000000000002},"last":0.53700000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574873,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":150965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":4.6159999999999997,"5min":4.6159999999999997,"15min":4.6159999999999997},"last":0.62},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.55000000000000004}]},{"osd":1,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":3.6600000000000001,"5min":3.6600000000000001,"15min":3.6600000000000001},"last":0.45000000000000001}]},{"osd":2,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":5.7160000000000002,"5min":5.7160000000000002,"15min":5.7160000000000002},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.76300000000000001}]},{"osd":3,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":3.7549999999999999,"5min":3.7549999999999999,"15min":3.7549999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.744,"5min":1.744,"15min":1.744},"last":0.73599999999999999}]},{"osd":5,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.79400000000000004}]},{"osd":6,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.93200000000000005}]},{"osd":7,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.89300000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574873,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":145614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.52500000000000002}]},{"osd":1,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.55000000000000004}]},{"osd":2,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.434}]},{"osd":3,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.623}]},{"osd":4,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.57299999999999995}]},{"osd":6,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.51500000000000001}]},{"osd":7,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1,"5min":1,"15min":1},"last":0.78600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T12:00:00.491 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T12:00:00.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.610+0000 7ff23519d700 1 Processor -- start 2023-10-14T12:00:00.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.610+0000 7ff23519d700 1 -- start start 2023-10-14T12:00:00.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.611+0000 7ff23519d700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff2300c1be0 0x7ff23014f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:00.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.611+0000 7ff23519d700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff23014fd80 0x7ff2301526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:00.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.611+0000 7ff23519d700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff230155c90 con 0x7ff23006f7b0 2023-10-14T12:00:00.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.611+0000 7ff23519d700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff230155f70 con 0x7ff2300c1be0 2023-10-14T12:00:00.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.611+0000 7ff23519d700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff230156250 con 0x7ff23014fd80 2023-10-14T12:00:00.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.611+0000 7ff22e59c700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff2300c1be0 0x7ff23014f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:00.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.611+0000 7ff22e59c700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff2300c1be0 0x7ff23014f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47142/0 (socket says 172.21.15.78:47142) 2023-10-14T12:00:00.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.611+0000 7ff22ed9d700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7ff23006f7b0 0x7ff230072360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52018/0 (socket says 172.21.15.78:52018) 2023-10-14T12:00:00.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.611+0000 7ff22e59c700 1 -- 172.21.15.78:0/239935850 learned_addr learned my addr 172.21.15.78:0/239935850 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:00.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.611+0000 7ff22f59e700 1 --2- 172.21.15.78:0/239935850 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff23014fd80 0x7ff2301526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:00.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.611+0000 7ff22e59c700 1 -- 172.21.15.78:0/239935850 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff23014fd80 msgr2=0x7ff2301526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T12:00:00.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.611+0000 7ff22e59c700 1 --2- 172.21.15.78:0/239935850 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff23014fd80 0x7ff2301526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:00.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.611+0000 7ff22e59c700 1 -- 172.21.15.78:0/239935850 >> v1:172.21.15.78:6789/0 conn(0x7ff23006f7b0 legacy=0x7ff230072360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:00.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.611+0000 7ff22dd9b700 1 -- 172.21.15.78:0/239935850 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3851122138 0 0) 0x7ff230155c90 con 0x7ff23006f7b0 2023-10-14T12:00:00.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.611+0000 7ff22e59c700 1 -- 172.21.15.78:0/239935850 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff230156530 con 0x7ff2300c1be0 2023-10-14T12:00:00.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.611+0000 7ff22e59c700 1 --2- 172.21.15.78:0/239935850 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2300c1be0 0x7ff23014f840 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7ff2200098f0 tx=0x7ff220009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac6e5d2dd198375c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:00.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.611+0000 7ff22dd9b700 1 -- 172.21.15.78:0/239935850 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22000b8a0 con 0x7ff2300c1be0 2023-10-14T12:00:00.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.611+0000 7ff22dd9b700 1 -- 172.21.15.78:0/239935850 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff22000ba00 con 0x7ff2300c1be0 2023-10-14T12:00:00.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.611+0000 7ff22dd9b700 1 -- 172.21.15.78:0/239935850 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22000bd90 con 0x7ff2300c1be0 2023-10-14T12:00:00.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.612+0000 7ff23519d700 1 -- 172.21.15.78:0/239935850 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2300c1be0 msgr2=0x7ff23014f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:00.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.612+0000 7ff23519d700 1 --2- 172.21.15.78:0/239935850 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2300c1be0 0x7ff23014f840 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7ff2200098f0 tx=0x7ff220009c00 comp rx=0 tx=0).stop 2023-10-14T12:00:00.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.612+0000 7ff23519d700 1 -- 172.21.15.78:0/239935850 shutdown_connections 2023-10-14T12:00:00.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.612+0000 7ff23519d700 1 --2- 172.21.15.78:0/239935850 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff23014fd80 0x7ff2301526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:00.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.612+0000 7ff23519d700 1 --2- 172.21.15.78:0/239935850 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2300c1be0 0x7ff23014f840 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:00.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.612+0000 7ff23519d700 1 -- 172.21.15.78:0/239935850 >> 172.21.15.78:0/239935850 conn(0x7ff2300c3ac0 msgr2=0x7ff2300c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:00.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.612+0000 7ff23519d700 1 -- 172.21.15.78:0/239935850 shutdown_connections 2023-10-14T12:00:00.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.612+0000 7ff23519d700 1 -- 172.21.15.78:0/239935850 wait complete. 2023-10-14T12:00:00.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.613+0000 7ff23519d700 1 Processor -- start 2023-10-14T12:00:00.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.613+0000 7ff23519d700 1 -- start start 2023-10-14T12:00:00.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.614+0000 7ff23519d700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff23006f7b0 0x7ff23014b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:00.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.614+0000 7ff23519d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff2300c1be0 0x7ff23014c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:00.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.614+0000 7ff23519d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff23014fd80 0x7ff2301eb070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:00.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.614+0000 7ff23519d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff23014bd00 con 0x7ff23014fd80 2023-10-14T12:00:00.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.614+0000 7ff23519d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff230143ab0 con 0x7ff2300c1be0 2023-10-14T12:00:00.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.614+0000 7ff23519d700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2301437d0 con 0x7ff23006f7b0 2023-10-14T12:00:00.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.614+0000 7ff22f59e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff23014fd80 0x7ff2301eb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:00.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.614+0000 7ff22f59e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff23014fd80 0x7ff2301eb070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47152/0 (socket says 172.21.15.78:47152) 2023-10-14T12:00:00.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.614+0000 7ff22f59e700 1 -- 172.21.15.78:0/3724120014 learned_addr learned my addr 172.21.15.78:0/3724120014 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:00.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.614+0000 7ff22ed9d700 1 --2- 172.21.15.78:0/3724120014 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff23006f7b0 0x7ff23014b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:00.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.614+0000 7ff22e59c700 1 --2- 172.21.15.78:0/3724120014 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff2300c1be0 0x7ff23014c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:00.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.614+0000 7ff22f59e700 1 -- 172.21.15.78:0/3724120014 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff23006f7b0 msgr2=0x7ff23014b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:00.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.615+0000 7ff22f59e700 1 --2- 172.21.15.78:0/3724120014 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff23006f7b0 0x7ff23014b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:00.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.615+0000 7ff22f59e700 1 -- 172.21.15.78:0/3724120014 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff2300c1be0 msgr2=0x7ff23014c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:00.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.615+0000 7ff22f59e700 1 --2- 172.21.15.78:0/3724120014 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff2300c1be0 0x7ff23014c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:00.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.615+0000 7ff22f59e700 1 -- 172.21.15.78:0/3724120014 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff220009550 con 0x7ff23014fd80 2023-10-14T12:00:00.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.615+0000 7ff22e59c700 1 --2- 172.21.15.78:0/3724120014 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff2300c1be0 0x7ff23014c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T12:00:00.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.615+0000 7ff22ed9d700 1 --2- 172.21.15.78:0/3724120014 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff23006f7b0 0x7ff23014b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T12:00:00.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.615+0000 7ff22f59e700 1 --2- 172.21.15.78:0/3724120014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff23014fd80 0x7ff2301eb070 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ff228007e60 tx=0x7ff22800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:00.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.615+0000 7ff2177fe700 1 -- 172.21.15.78:0/3724120014 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22800ede0 con 0x7ff23014fd80 2023-10-14T12:00:00.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.615+0000 7ff2177fe700 1 -- 172.21.15.78:0/3724120014 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff228014420 con 0x7ff23014fd80 2023-10-14T12:00:00.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.615+0000 7ff23519d700 1 -- 172.21.15.78:0/3724120014 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff230155030 con 0x7ff23014fd80 2023-10-14T12:00:00.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.615+0000 7ff2177fe700 1 -- 172.21.15.78:0/3724120014 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2280147d0 con 0x7ff23014fd80 2023-10-14T12:00:00.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.615+0000 7ff23519d700 1 -- 172.21.15.78:0/3724120014 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2301555f0 con 0x7ff23014fd80 2023-10-14T12:00:00.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.616+0000 7ff23519d700 1 -- 172.21.15.78:0/3724120014 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff230074b80 con 0x7ff23014fd80 2023-10-14T12:00:00.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.616+0000 7ff2177fe700 1 -- 172.21.15.78:0/3724120014 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff228014930 con 0x7ff23014fd80 2023-10-14T12:00:00.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.617+0000 7ff2177fe700 1 --2- 172.21.15.78:0/3724120014 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff21c06af00 0x7ff21c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:00.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.617+0000 7ff2177fe700 1 -- 172.21.15.78:0/3724120014 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff228089970 con 0x7ff23014fd80 2023-10-14T12:00:00.633 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.617+0000 7ff22ed9d700 1 --2- 172.21.15.78:0/3724120014 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff21c06af00 0x7ff21c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:00.633 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.617+0000 7ff22ed9d700 1 --2- 172.21.15.78:0/3724120014 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff21c06af00 0x7ff21c06d3c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff224000f60 tx=0x7ff224009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:00.633 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.621+0000 7ff2177fe700 1 -- 172.21.15.78:0/3724120014 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff228054db0 con 0x7ff23014fd80 2023-10-14T12:00:00.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.805+0000 7ff23519d700 1 -- 172.21.15.78:0/3724120014 --> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff230156810 con 0x7ff21c06af00 2023-10-14T12:00:00.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.808+0000 7ff2177fe700 1 -- 172.21.15.78:0/3724120014 <== mgr.4104 v2:172.21.15.78:6832/3171890074 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+51053 (secure 0 0 0) 0x7ff224001cc0 con 0x7ff21c06af00 2023-10-14T12:00:00.809 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T12:00:00.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.810+0000 7ff23519d700 1 -- 172.21.15.78:0/3724120014 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff21c06af00 msgr2=0x7ff21c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:00.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.810+0000 7ff23519d700 1 --2- 172.21.15.78:0/3724120014 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff21c06af00 0x7ff21c06d3c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff224000f60 tx=0x7ff224009040 comp rx=0 tx=0).stop 2023-10-14T12:00:00.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.810+0000 7ff23519d700 1 -- 172.21.15.78:0/3724120014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff23014fd80 msgr2=0x7ff2301eb070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:00.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.810+0000 7ff23519d700 1 --2- 172.21.15.78:0/3724120014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff23014fd80 0x7ff2301eb070 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ff228007e60 tx=0x7ff22800e3c0 comp rx=0 tx=0).stop 2023-10-14T12:00:00.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.810+0000 7ff23519d700 1 -- 172.21.15.78:0/3724120014 shutdown_connections 2023-10-14T12:00:00.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.810+0000 7ff23519d700 1 --2- 172.21.15.78:0/3724120014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff23014fd80 0x7ff2301eb070 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:00.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.810+0000 7ff23519d700 1 --2- 172.21.15.78:0/3724120014 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff23006f7b0 0x7ff23014b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:00.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.810+0000 7ff23519d700 1 --2- 172.21.15.78:0/3724120014 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff21c06af00 0x7ff21c06d3c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:00.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.810+0000 7ff23519d700 1 --2- 172.21.15.78:0/3724120014 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff2300c1be0 0x7ff23014c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:00.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.810+0000 7ff23519d700 1 -- 172.21.15.78:0/3724120014 >> 172.21.15.78:0/3724120014 conn(0x7ff2300c3ac0 msgr2=0x7ff2300667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:00.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.810+0000 7ff23519d700 1 -- 172.21.15.78:0/3724120014 shutdown_connections 2023-10-14T12:00:00.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.810+0000 7ff23519d700 1 -- 172.21.15.78:0/3724120014 wait complete. 2023-10-14T12:00:00.815 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2023-10-14T12:00:00.822 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2023-10-14T11:59:58.989175+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219296,"kb_used_data":3124,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755424,"statfs":{"total":773094113280,"available":772869554176,"internally_reserved":0,"allocated":3198976,"data_stored":2361333,"data_compressed":65244,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2410,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-45,"num_write_kb":-14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001208"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":175,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T11:59:55.540323+0000","last_change":"2023-10-14T11:58:10.616726+0000","last_active":"2023-10-14T11:59:55.540323+0000","last_peered":"2023-10-14T11:59:55.540323+0000","last_clean":"2023-10-14T11:59:55.540323+0000","last_became_active":"2023-10-14T11:58:10.615628+0000","last_became_peered":"2023-10-14T11:58:10.615628+0000","last_unstale":"2023-10-14T11:59:55.540323+0000","last_undegraded":"2023-10-14T11:59:55.540323+0000","last_fullsized":"2023-10-14T11:59:55.540323+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:09.001265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:09.001265+0000","last_clean_scrub_stamp":"2023-10-14T11:58:09.001265+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:22:50.496050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65244,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574873,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":454656,"data_stored":741333,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.441}]},{"osd":1,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.317},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.35899999999999999}]},{"osd":2,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":2.8250000000000002,"5min":2.8250000000000002,"15min":2.8250000000000002},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.46300000000000002}]},{"osd":3,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":2.8639999999999999,"5min":2.8639999999999999,"15min":2.8639999999999999},"last":0.66800000000000004}]},{"osd":4,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.47699999999999998}]},{"osd":5,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.498},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.57099999999999995}]},{"osd":6,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.54800000000000004}]}]},{"osd":6,"up_from":13,"seq":55834574873,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":145558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.57099999999999995}]},{"osd":1,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.61399999999999999}]},{"osd":2,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.54600000000000004}]},{"osd":3,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":2.2789999999999999,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.496}]},{"osd":4,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.43099999999999999}]},{"osd":5,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.47999999999999998}]},{"osd":7,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.65700000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574873,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":146576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.30599999999999999}]},{"osd":2,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.37}]},{"osd":3,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.495,"5min":0.495,"15min":0.495},"last":0.34599999999999997}]},{"osd":4,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.72199999999999998}]},{"osd":5,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.57399999999999995}]},{"osd":6,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.68300000000000005}]},{"osd":7,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.58799999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574872,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27448,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344392,"statfs":{"total":96636764160,"available":96608657408,"internally_reserved":0,"allocated":438272,"data_stored":735310,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.31,"15min":0.31},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"last":0.26500000000000001}]},{"osd":2,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.378,"5min":0.378,"15min":0.378},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.318}]},{"osd":3,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.439,"5min":0.439,"15min":0.439},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.41799999999999998}]},{"osd":4,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.53400000000000003}]},{"osd":5,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.55600000000000005}]},{"osd":6,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.52100000000000002}]},{"osd":7,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.47099999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574872,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":376832,"data_stored":145012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 11:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.28799999999999998}]},{"osd":1,"last update":"Sat Oct 14 11:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.37},{"interface":"front","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.35599999999999998}]},{"osd":3,"last update":"Sat Oct 14 11:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.33100000000000002}]},{"osd":4,"last update":"Sat Oct 14 11:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1,"5min":1,"15min":1},"last":0.437},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.57999999999999996}]},{"osd":5,"last update":"Sat Oct 14 11:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.48199999999999998}]},{"osd":6,"last update":"Sat Oct 14 11:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.42399999999999999}]},{"osd":7,"last update":"Sat Oct 14 11:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.622},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.44600000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":150965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 11:59:10 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.226,"5min":7.226,"15min":7.226},"last":0.25600000000000001},{"interface":"front","average":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":21.067,"5min":21.067,"15min":21.067},"last":0.30499999999999999}]},{"osd":1,"last update":"Sat Oct 14 11:59:10 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":7.3419999999999996,"5min":7.3419999999999996,"15min":7.3419999999999996},"last":0.36599999999999999}]},{"osd":2,"last update":"Sat Oct 14 11:59:10 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":7.3120000000000003,"5min":7.3120000000000003,"15min":7.3120000000000003},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":7.1870000000000003,"5min":7.1870000000000003,"15min":7.1870000000000003},"last":0.32800000000000001}]},{"osd":4,"last update":"Sat Oct 14 11:59:10 2023","interfaces":[{"interface":"back","average":{"1min":1.8640000000000001,"5min":1.8640000000000001,"15min":1.8640000000000001},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":20.984000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":17.902999999999999,"5min":17.902999999999999,"15min":17.902999999999999},"last":0.56799999999999995}]},{"osd":5,"last update":"Sat Oct 14 11:59:10 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":7.2560000000000002,"5min":7.2560000000000002,"15min":7.2560000000000002},"last":0.55100000000000005},{"interface":"front","average":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":21.100000000000001,"5min":21.100000000000001,"15min":21.100000000000001},"last":0.46500000000000002}]},{"osd":6,"last update":"Sat Oct 14 11:59:10 2023","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.002,"15min":1.002},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":7.1589999999999998,"5min":7.1589999999999998,"15min":7.1589999999999998},"last":0.47399999999999998},{"interface":"front","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"last":0.58999999999999997}]},{"osd":7,"last update":"Sat Oct 14 11:59:10 2023","interfaces":[{"interface":"back","average":{"1min":1.891,"5min":1.891,"15min":1.891},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":21.035,"5min":21.035,"15min":21.035},"last":0.48599999999999999},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":7.1230000000000002,"5min":7.1230000000000002,"15min":7.1230000000000002},"last":0.53700000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574873,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":150965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":4.6159999999999997,"5min":4.6159999999999997,"15min":4.6159999999999997},"last":0.62},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.55000000000000004}]},{"osd":1,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":3.6600000000000001,"5min":3.6600000000000001,"15min":3.6600000000000001},"last":0.45000000000000001}]},{"osd":2,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":5.7160000000000002,"5min":5.7160000000000002,"15min":5.7160000000000002},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.76300000000000001}]},{"osd":3,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":3.7549999999999999,"5min":3.7549999999999999,"15min":3.7549999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.744,"5min":1.744,"15min":1.744},"last":0.73599999999999999}]},{"osd":5,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.79400000000000004}]},{"osd":6,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.93200000000000005}]},{"osd":7,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.89300000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574873,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":145614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.52500000000000002}]},{"osd":1,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.55000000000000004}]},{"osd":2,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.434}]},{"osd":3,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.623}]},{"osd":4,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.57299999999999995}]},{"osd":6,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.51500000000000001}]},{"osd":7,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1,"5min":1,"15min":1},"last":0.78600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T12:00:00.824 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-14T12:00:00.825 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T12:00:00.945 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.944+0000 7ff7ee981700 1 Processor -- start 2023-10-14T12:00:00.945 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.944+0000 7ff7ee981700 1 -- start start 2023-10-14T12:00:00.947 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.945+0000 7ff7ee981700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff7e8157e60 0x7ff7e806e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:00.947 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.945+0000 7ff7ee981700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7e8155590 con 0x7ff7e813dd60 2023-10-14T12:00:00.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.945+0000 7ff7ee981700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7e8154980 con 0x7ff7e8071830 2023-10-14T12:00:00.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.945+0000 7ff7ee981700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7e8154c60 con 0x7ff7e8157e60 2023-10-14T12:00:00.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.945+0000 7ff7e77fe700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff7e8157e60 0x7ff7e806e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:00.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.945+0000 7ff7e7fff700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7ff7e813dd60 0x7ff7e806a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52032/0 (socket says 172.21.15.78:52032) 2023-10-14T12:00:00.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.945+0000 7ff7e7fff700 1 -- 172.21.15.78:0/3778269297 learned_addr learned my addr 172.21.15.78:0/3778269297 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T12:00:00.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.945+0000 7ff7e77fe700 1 -- 172.21.15.78:0/3778269297 >> v1:172.21.15.195:6789/0 conn(0x7ff7e8071830 legacy=0x7ff7e806ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T12:00:00.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.945+0000 7ff7e6ffd700 1 -- 172.21.15.78:0/3778269297 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 904406279 0 0) 0x7ff7e8155590 con 0x7ff7e813dd60 2023-10-14T12:00:00.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.945+0000 7ff7e77fe700 1 -- 172.21.15.78:0/3778269297 >> v1:172.21.15.78:6789/0 conn(0x7ff7e813dd60 legacy=0x7ff7e806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:00.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.945+0000 7ff7e77fe700 1 -- 172.21.15.78:0/3778269297 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7e8154f40 con 0x7ff7e8157e60 2023-10-14T12:00:00.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.945+0000 7ff7e77fe700 1 --2- 172.21.15.78:0/3778269297 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7e8157e60 0x7ff7e806e7c0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc009840 tx=0x7ff7dc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58d9e06f2cd55078 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:00.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.945+0000 7ff7e6ffd700 1 -- 172.21.15.78:0/3778269297 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7dc004230 con 0x7ff7e8157e60 2023-10-14T12:00:00.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.945+0000 7ff7e6ffd700 1 -- 172.21.15.78:0/3778269297 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7dc004390 con 0x7ff7e8157e60 2023-10-14T12:00:00.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.945+0000 7ff7e6ffd700 1 -- 172.21.15.78:0/3778269297 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7dc0036a0 con 0x7ff7e8157e60 2023-10-14T12:00:00.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.946+0000 7ff7ee981700 1 -- 172.21.15.78:0/3778269297 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7e8157e60 msgr2=0x7ff7e806e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:00.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.946+0000 7ff7ee981700 1 --2- 172.21.15.78:0/3778269297 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7e8157e60 0x7ff7e806e7c0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc009840 tx=0x7ff7dc009b50 comp rx=0 tx=0).stop 2023-10-14T12:00:00.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.946+0000 7ff7ee981700 1 -- 172.21.15.78:0/3778269297 shutdown_connections 2023-10-14T12:00:00.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.946+0000 7ff7ee981700 1 --2- 172.21.15.78:0/3778269297 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7e8157e60 0x7ff7e806e7c0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:00.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.946+0000 7ff7ee981700 1 -- 172.21.15.78:0/3778269297 >> 172.21.15.78:0/3778269297 conn(0x7ff7e80c6fc0 msgr2=0x7ff7e80c73d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:00.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.946+0000 7ff7ee981700 1 -- 172.21.15.78:0/3778269297 shutdown_connections 2023-10-14T12:00:00.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.946+0000 7ff7ee981700 1 -- 172.21.15.78:0/3778269297 wait complete. 2023-10-14T12:00:00.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.947+0000 7ff7ee981700 1 Processor -- start 2023-10-14T12:00:00.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.947+0000 7ff7ee981700 1 -- start start 2023-10-14T12:00:00.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.948+0000 7ff7ee981700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff7e8071830 0x7ff7e81ed9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:00.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.948+0000 7ff7ee981700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7e813dd60 0x7ff7e806e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:00.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.948+0000 7ff7ee981700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff7e8157e60 0x7ff7e8135a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:00.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.948+0000 7ff7ee981700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e80c6b30 con 0x7ff7e813dd60 2023-10-14T12:00:00.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.948+0000 7ff7ee981700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e80c59d0 con 0x7ff7e8157e60 2023-10-14T12:00:00.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.948+0000 7ff7ee981700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e80c6790 con 0x7ff7e8071830 2023-10-14T12:00:00.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.948+0000 7ff7e77fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7e813dd60 0x7ff7e806e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:00.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.948+0000 7ff7e77fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7e813dd60 0x7ff7e806e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47164/0 (socket says 172.21.15.78:47164) 2023-10-14T12:00:00.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.948+0000 7ff7e77fe700 1 -- 172.21.15.78:0/975911305 learned_addr learned my addr 172.21.15.78:0/975911305 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:00.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.948+0000 7ff7ecf1e700 1 --2- 172.21.15.78:0/975911305 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff7e8157e60 0x7ff7e8135a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:00.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.948+0000 7ff7e7fff700 1 --2- 172.21.15.78:0/975911305 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff7e8071830 0x7ff7e81ed9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:00.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.948+0000 7ff7e77fe700 1 -- 172.21.15.78:0/975911305 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff7e8071830 msgr2=0x7ff7e81ed9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:00.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.948+0000 7ff7e77fe700 1 --2- 172.21.15.78:0/975911305 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff7e8071830 0x7ff7e81ed9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:00.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.948+0000 7ff7e77fe700 1 -- 172.21.15.78:0/975911305 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff7e8157e60 msgr2=0x7ff7e8135a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:00.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.948+0000 7ff7e77fe700 1 --2- 172.21.15.78:0/975911305 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff7e8157e60 0x7ff7e8135a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:00.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.948+0000 7ff7e77fe700 1 -- 172.21.15.78:0/975911305 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7dc0094a0 con 0x7ff7e813dd60 2023-10-14T12:00:00.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.948+0000 7ff7ecf1e700 1 --2- 172.21.15.78:0/975911305 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff7e8157e60 0x7ff7e8135a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T12:00:00.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.948+0000 7ff7e77fe700 1 --2- 172.21.15.78:0/975911305 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7e813dd60 0x7ff7e806e980 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc0090b0 tx=0x7ff7dc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:00.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.949+0000 7ff7e4ff9700 1 -- 172.21.15.78:0/975911305 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7dc011830 con 0x7ff7e813dd60 2023-10-14T12:00:00.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.949+0000 7ff7e4ff9700 1 -- 172.21.15.78:0/975911305 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7dc01e070 con 0x7ff7e813dd60 2023-10-14T12:00:00.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.949+0000 7ff7e4ff9700 1 -- 172.21.15.78:0/975911305 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7dc011bf0 con 0x7ff7e813dd60 2023-10-14T12:00:00.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.949+0000 7ff7ee981700 1 -- 172.21.15.78:0/975911305 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7e80c56f0 con 0x7ff7e813dd60 2023-10-14T12:00:00.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.949+0000 7ff7ee981700 1 -- 172.21.15.78:0/975911305 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7e80c53d0 con 0x7ff7e813dd60 2023-10-14T12:00:00.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.950+0000 7ff7ee981700 1 -- 172.21.15.78:0/975911305 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7e8074b80 con 0x7ff7e813dd60 2023-10-14T12:00:00.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.950+0000 7ff7e4ff9700 1 -- 172.21.15.78:0/975911305 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff7dc011d50 con 0x7ff7e813dd60 2023-10-14T12:00:00.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.950+0000 7ff7e4ff9700 1 --2- 172.21.15.78:0/975911305 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff7d406af00 0x7ff7d406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:00.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.951+0000 7ff7e4ff9700 1 -- 172.21.15.78:0/975911305 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff7dc08a5d0 con 0x7ff7e813dd60 2023-10-14T12:00:00.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.951+0000 7ff7e7fff700 1 --2- 172.21.15.78:0/975911305 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff7d406af00 0x7ff7d406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:00.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.951+0000 7ff7e7fff700 1 --2- 172.21.15.78:0/975911305 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff7d406af00 0x7ff7d406d3c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff7d8000f60 tx=0x7ff7d8007c00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:00.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:00.955+0000 7ff7e4ff9700 1 -- 172.21.15.78:0/975911305 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff7dc055a10 con 0x7ff7e813dd60 2023-10-14T12:00:01.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.140+0000 7ff7ee981700 1 -- 172.21.15.78:0/975911305 --> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff7e8155220 con 0x7ff7d406af00 2023-10-14T12:00:01.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.142+0000 7ff7e4ff9700 1 -- 172.21.15.78:0/975911305 <== mgr.4104 v2:172.21.15.78:6832/3171890074 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+51053 (secure 0 0 0) 0x7ff7d80012b0 con 0x7ff7d406af00 2023-10-14T12:00:01.144 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T12:00:01.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.144+0000 7ff7ee981700 1 -- 172.21.15.78:0/975911305 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff7d406af00 msgr2=0x7ff7d406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:01.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.144+0000 7ff7ee981700 1 --2- 172.21.15.78:0/975911305 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff7d406af00 0x7ff7d406d3c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff7d8000f60 tx=0x7ff7d8007c00 comp rx=0 tx=0).stop 2023-10-14T12:00:01.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.144+0000 7ff7ee981700 1 -- 172.21.15.78:0/975911305 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7e813dd60 msgr2=0x7ff7e806e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:01.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.144+0000 7ff7ee981700 1 --2- 172.21.15.78:0/975911305 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7e813dd60 0x7ff7e806e980 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc0090b0 tx=0x7ff7dc004750 comp rx=0 tx=0).stop 2023-10-14T12:00:01.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.144+0000 7ff7ee981700 1 -- 172.21.15.78:0/975911305 shutdown_connections 2023-10-14T12:00:01.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.144+0000 7ff7ee981700 1 --2- 172.21.15.78:0/975911305 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff7d406af00 0x7ff7d406d3c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:01.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.144+0000 7ff7ee981700 1 --2- 172.21.15.78:0/975911305 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff7e8157e60 0x7ff7e8135a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:01.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.144+0000 7ff7ee981700 1 --2- 172.21.15.78:0/975911305 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff7e8071830 0x7ff7e81ed9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:01.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.144+0000 7ff7ee981700 1 --2- 172.21.15.78:0/975911305 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7e813dd60 0x7ff7e806e980 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:01.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.144+0000 7ff7ee981700 1 -- 172.21.15.78:0/975911305 >> 172.21.15.78:0/975911305 conn(0x7ff7e80c6fc0 msgr2=0x7ff7e813adc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:01.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.144+0000 7ff7ee981700 1 -- 172.21.15.78:0/975911305 shutdown_connections 2023-10-14T12:00:01.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.144+0000 7ff7ee981700 1 -- 172.21.15.78:0/975911305 wait complete. 2023-10-14T12:00:01.150 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2023-10-14T12:00:01.158 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2023-10-14T12:00:00.989458+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219296,"kb_used_data":3124,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755424,"statfs":{"total":773094113280,"available":772869554176,"internally_reserved":0,"allocated":3198976,"data_stored":2361333,"data_compressed":65244,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2410,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-45,"num_write_kb":-14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.492289"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":175,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T11:59:55.540323+0000","last_change":"2023-10-14T11:58:10.616726+0000","last_active":"2023-10-14T11:59:55.540323+0000","last_peered":"2023-10-14T11:59:55.540323+0000","last_clean":"2023-10-14T11:59:55.540323+0000","last_became_active":"2023-10-14T11:58:10.615628+0000","last_became_peered":"2023-10-14T11:58:10.615628+0000","last_unstale":"2023-10-14T11:59:55.540323+0000","last_undegraded":"2023-10-14T11:59:55.540323+0000","last_fullsized":"2023-10-14T11:59:55.540323+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:09.001265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:09.001265+0000","last_clean_scrub_stamp":"2023-10-14T11:58:09.001265+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:22:50.496050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65244,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574873,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":454656,"data_stored":741333,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.441}]},{"osd":1,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.317},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.35899999999999999}]},{"osd":2,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":2.8250000000000002,"5min":2.8250000000000002,"15min":2.8250000000000002},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.46300000000000002}]},{"osd":3,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":2.8639999999999999,"5min":2.8639999999999999,"15min":2.8639999999999999},"last":0.66800000000000004}]},{"osd":4,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.47699999999999998}]},{"osd":5,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.498},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.57099999999999995}]},{"osd":6,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.54800000000000004}]}]},{"osd":6,"up_from":13,"seq":55834574873,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":145558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.57099999999999995}]},{"osd":1,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.61399999999999999}]},{"osd":2,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.54600000000000004}]},{"osd":3,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":2.2789999999999999,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.496}]},{"osd":4,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.43099999999999999}]},{"osd":5,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.47999999999999998}]},{"osd":7,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.65700000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574873,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":146576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.30599999999999999}]},{"osd":2,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.37}]},{"osd":3,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.495,"5min":0.495,"15min":0.495},"last":0.34599999999999997}]},{"osd":4,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.72199999999999998}]},{"osd":5,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.57399999999999995}]},{"osd":6,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.68300000000000005}]},{"osd":7,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.58799999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574872,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27448,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344392,"statfs":{"total":96636764160,"available":96608657408,"internally_reserved":0,"allocated":438272,"data_stored":735310,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.31,"15min":0.31},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"last":0.26500000000000001}]},{"osd":2,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.378,"5min":0.378,"15min":0.378},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.318}]},{"osd":3,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.439,"5min":0.439,"15min":0.439},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.41799999999999998}]},{"osd":4,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.53400000000000003}]},{"osd":5,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.55600000000000005}]},{"osd":6,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.52100000000000002}]},{"osd":7,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.47099999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574872,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":376832,"data_stored":145012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 11:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.28799999999999998}]},{"osd":1,"last update":"Sat Oct 14 11:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.37},{"interface":"front","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.35599999999999998}]},{"osd":3,"last update":"Sat Oct 14 11:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.33100000000000002}]},{"osd":4,"last update":"Sat Oct 14 11:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1,"5min":1,"15min":1},"last":0.437},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.57999999999999996}]},{"osd":5,"last update":"Sat Oct 14 11:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.48199999999999998}]},{"osd":6,"last update":"Sat Oct 14 11:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.42399999999999999}]},{"osd":7,"last update":"Sat Oct 14 11:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.622},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.44600000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":150965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 11:59:10 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.226,"5min":7.226,"15min":7.226},"last":0.25600000000000001},{"interface":"front","average":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":21.067,"5min":21.067,"15min":21.067},"last":0.30499999999999999}]},{"osd":1,"last update":"Sat Oct 14 11:59:10 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":7.3419999999999996,"5min":7.3419999999999996,"15min":7.3419999999999996},"last":0.36599999999999999}]},{"osd":2,"last update":"Sat Oct 14 11:59:10 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":7.3120000000000003,"5min":7.3120000000000003,"15min":7.3120000000000003},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":7.1870000000000003,"5min":7.1870000000000003,"15min":7.1870000000000003},"last":0.32800000000000001}]},{"osd":4,"last update":"Sat Oct 14 11:59:10 2023","interfaces":[{"interface":"back","average":{"1min":1.8640000000000001,"5min":1.8640000000000001,"15min":1.8640000000000001},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":20.984000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":17.902999999999999,"5min":17.902999999999999,"15min":17.902999999999999},"last":0.56799999999999995}]},{"osd":5,"last update":"Sat Oct 14 11:59:10 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":7.2560000000000002,"5min":7.2560000000000002,"15min":7.2560000000000002},"last":0.55100000000000005},{"interface":"front","average":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":21.100000000000001,"5min":21.100000000000001,"15min":21.100000000000001},"last":0.46500000000000002}]},{"osd":6,"last update":"Sat Oct 14 11:59:10 2023","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.002,"15min":1.002},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":7.1589999999999998,"5min":7.1589999999999998,"15min":7.1589999999999998},"last":0.47399999999999998},{"interface":"front","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"last":0.58999999999999997}]},{"osd":7,"last update":"Sat Oct 14 11:59:10 2023","interfaces":[{"interface":"back","average":{"1min":1.891,"5min":1.891,"15min":1.891},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":21.035,"5min":21.035,"15min":21.035},"last":0.48599999999999999},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":7.1230000000000002,"5min":7.1230000000000002,"15min":7.1230000000000002},"last":0.53700000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574873,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":150965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":4.6159999999999997,"5min":4.6159999999999997,"15min":4.6159999999999997},"last":0.62},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.55000000000000004}]},{"osd":1,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":3.6600000000000001,"5min":3.6600000000000001,"15min":3.6600000000000001},"last":0.45000000000000001}]},{"osd":2,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":5.7160000000000002,"5min":5.7160000000000002,"15min":5.7160000000000002},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.76300000000000001}]},{"osd":3,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":3.7549999999999999,"5min":3.7549999999999999,"15min":3.7549999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.744,"5min":1.744,"15min":1.744},"last":0.73599999999999999}]},{"osd":5,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.79400000000000004}]},{"osd":6,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.93200000000000005}]},{"osd":7,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.89300000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574873,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":145614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.52500000000000002}]},{"osd":1,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.55000000000000004}]},{"osd":2,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.434}]},{"osd":3,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.623}]},{"osd":4,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.57299999999999995}]},{"osd":6,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.51500000000000001}]},{"osd":7,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1,"5min":1,"15min":1},"last":0.78600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T12:00:01.160 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T12:00:01.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.279+0000 7fda12bb7700 1 Processor -- start 2023-10-14T12:00:01.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.279+0000 7fda12bb7700 1 -- start start 2023-10-14T12:00:01.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.280+0000 7fda12bb7700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fda0c14b6b0 0x7fda0c072580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:01.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.280+0000 7fda12bb7700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fda0c154bb0 0x7fda0c155040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:01.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.280+0000 7fda12bb7700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda0c0c25e0 con 0x7fda0c06db50 2023-10-14T12:00:01.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.280+0000 7fda12bb7700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda0c0a87b0 con 0x7fda0c154bb0 2023-10-14T12:00:01.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.280+0000 7fda12bb7700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda0c14ba90 con 0x7fda0c14b6b0 2023-10-14T12:00:01.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.280+0000 7fda11154700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fda0c154bb0 0x7fda0c155040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:01.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.280+0000 7fda0bfff700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fda0c06db50 0x7fda0c0c2a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52046/0 (socket says 172.21.15.78:52046) 2023-10-14T12:00:01.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.280+0000 7fda11154700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fda0c154bb0 0x7fda0c155040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47174/0 (socket says 172.21.15.78:47174) 2023-10-14T12:00:01.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.280+0000 7fda0bfff700 1 -- 172.21.15.78:0/4022387037 learned_addr learned my addr 172.21.15.78:0/4022387037 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T12:00:01.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.280+0000 7fda10953700 1 --2- 172.21.15.78:0/4022387037 >> v2:172.21.15.195:3300/0 conn(0x7fda0c14b6b0 0x7fda0c072580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:01.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.280+0000 7fda11154700 1 -- 172.21.15.78:0/4022387037 >> v2:172.21.15.195:3300/0 conn(0x7fda0c14b6b0 msgr2=0x7fda0c072580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T12:00:01.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.280+0000 7fda11154700 1 --2- 172.21.15.78:0/4022387037 >> v2:172.21.15.195:3300/0 conn(0x7fda0c14b6b0 0x7fda0c072580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:01.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.280+0000 7fda11154700 1 -- 172.21.15.78:0/4022387037 >> v1:172.21.15.78:6789/0 conn(0x7fda0c06db50 legacy=0x7fda0c0c2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:01.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.280+0000 7fda11154700 1 -- 172.21.15.78:0/4022387037 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda0c155660 con 0x7fda0c154bb0 2023-10-14T12:00:01.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.280+0000 7fda0b7fe700 1 -- 172.21.15.78:0/4022387037 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3166647501 0 0) 0x7fda0c0c25e0 con 0x7fda0c06db50 2023-10-14T12:00:01.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.280+0000 7fda11154700 1 --2- 172.21.15.78:0/4022387037 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda0c154bb0 0x7fda0c155040 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fda0c0c13b0 tx=0x7fda0400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91f6fcde94cd5515 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:01.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.280+0000 7fda0b7fe700 1 -- 172.21.15.78:0/4022387037 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0400e070 con 0x7fda0c154bb0 2023-10-14T12:00:01.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.280+0000 7fda0b7fe700 1 -- 172.21.15.78:0/4022387037 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda0400bbd0 con 0x7fda0c154bb0 2023-10-14T12:00:01.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.280+0000 7fda0b7fe700 1 -- 172.21.15.78:0/4022387037 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda04012430 con 0x7fda0c154bb0 2023-10-14T12:00:01.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.281+0000 7fda12bb7700 1 -- 172.21.15.78:0/4022387037 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda0c154bb0 msgr2=0x7fda0c155040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:01.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.281+0000 7fda12bb7700 1 --2- 172.21.15.78:0/4022387037 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda0c154bb0 0x7fda0c155040 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fda0c0c13b0 tx=0x7fda0400b330 comp rx=0 tx=0).stop 2023-10-14T12:00:01.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.281+0000 7fda12bb7700 1 -- 172.21.15.78:0/4022387037 shutdown_connections 2023-10-14T12:00:01.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.281+0000 7fda12bb7700 1 --2- 172.21.15.78:0/4022387037 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda0c154bb0 0x7fda0c155040 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:01.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.281+0000 7fda12bb7700 1 --2- 172.21.15.78:0/4022387037 >> v2:172.21.15.195:3300/0 conn(0x7fda0c14b6b0 0x7fda0c072580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:01.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.281+0000 7fda12bb7700 1 -- 172.21.15.78:0/4022387037 >> 172.21.15.78:0/4022387037 conn(0x7fda0c0c3ac0 msgr2=0x7fda0c0c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:01.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.281+0000 7fda12bb7700 1 -- 172.21.15.78:0/4022387037 shutdown_connections 2023-10-14T12:00:01.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.281+0000 7fda12bb7700 1 -- 172.21.15.78:0/4022387037 wait complete. 2023-10-14T12:00:01.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.282+0000 7fda12bb7700 1 Processor -- start 2023-10-14T12:00:01.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.282+0000 7fda12bb7700 1 -- start start 2023-10-14T12:00:01.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.283+0000 7fda12bb7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda0c06db50 0x7fda0c0aafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:01.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.283+0000 7fda12bb7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda0c14b6b0 0x7fda0c0c6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:01.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.283+0000 7fda12bb7700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fda0c154bb0 0x7fda0c1e6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:01.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.283+0000 7fda12bb7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda0c1e5b50 con 0x7fda0c06db50 2023-10-14T12:00:01.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.283+0000 7fda12bb7700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda0c1e6730 con 0x7fda0c14b6b0 2023-10-14T12:00:01.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.283+0000 7fda12bb7700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda0c0aab30 con 0x7fda0c154bb0 2023-10-14T12:00:01.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.283+0000 7fda10953700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda0c06db50 0x7fda0c0aafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:01.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.283+0000 7fda10953700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda0c06db50 0x7fda0c0aafc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47178/0 (socket says 172.21.15.78:47178) 2023-10-14T12:00:01.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.283+0000 7fda10953700 1 -- 172.21.15.78:0/1947143157 learned_addr learned my addr 172.21.15.78:0/1947143157 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:01.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.283+0000 7fda11154700 1 --2- 172.21.15.78:0/1947143157 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fda0c154bb0 0x7fda0c1e6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:01.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.283+0000 7fda0bfff700 1 --2- 172.21.15.78:0/1947143157 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda0c14b6b0 0x7fda0c0c6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:01.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.283+0000 7fda10953700 1 -- 172.21.15.78:0/1947143157 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fda0c154bb0 msgr2=0x7fda0c1e6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:01.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.283+0000 7fda10953700 1 --2- 172.21.15.78:0/1947143157 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fda0c154bb0 0x7fda0c1e6bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:01.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.283+0000 7fda10953700 1 -- 172.21.15.78:0/1947143157 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda0c14b6b0 msgr2=0x7fda0c0c6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:01.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.283+0000 7fda10953700 1 --2- 172.21.15.78:0/1947143157 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda0c14b6b0 0x7fda0c0c6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:01.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.283+0000 7fda10953700 1 -- 172.21.15.78:0/1947143157 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda04009a70 con 0x7fda0c06db50 2023-10-14T12:00:01.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.283+0000 7fda10953700 1 --2- 172.21.15.78:0/1947143157 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda0c06db50 0x7fda0c0aafc0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fda0000e810 tx=0x7fda0000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:01.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.283+0000 7fda097fa700 1 -- 172.21.15.78:0/1947143157 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda00013610 con 0x7fda0c06db50 2023-10-14T12:00:01.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.284+0000 7fda097fa700 1 -- 172.21.15.78:0/1947143157 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda00013770 con 0x7fda0c06db50 2023-10-14T12:00:01.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.284+0000 7fda12bb7700 1 -- 172.21.15.78:0/1947143157 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda0c1e5870 con 0x7fda0c06db50 2023-10-14T12:00:01.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.284+0000 7fda097fa700 1 -- 172.21.15.78:0/1947143157 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda00013b20 con 0x7fda0c06db50 2023-10-14T12:00:01.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.284+0000 7fda12bb7700 1 -- 172.21.15.78:0/1947143157 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda0c0aa7c0 con 0x7fda0c06db50 2023-10-14T12:00:01.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.284+0000 7fda12bb7700 1 -- 172.21.15.78:0/1947143157 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda0c0aae10 con 0x7fda0c06db50 2023-10-14T12:00:01.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.285+0000 7fda097fa700 1 -- 172.21.15.78:0/1947143157 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fda00013c80 con 0x7fda0c06db50 2023-10-14T12:00:01.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.285+0000 7fda097fa700 1 --2- 172.21.15.78:0/1947143157 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd9f006af00 0x7fd9f006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:01.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.285+0000 7fda0bfff700 1 --2- 172.21.15.78:0/1947143157 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd9f006af00 0x7fd9f006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:01.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.285+0000 7fda097fa700 1 -- 172.21.15.78:0/1947143157 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fda00089020 con 0x7fda0c06db50 2023-10-14T12:00:01.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.286+0000 7fda0bfff700 1 --2- 172.21.15.78:0/1947143157 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd9f006af00 0x7fd9f006d3c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fda0c0c1170 tx=0x7fd9f8007be0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:01.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.290+0000 7fda097fa700 1 -- 172.21.15.78:0/1947143157 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fda000543e0 con 0x7fda0c06db50 2023-10-14T12:00:01.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.481+0000 7fda12bb7700 1 -- 172.21.15.78:0/1947143157 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda0c074b80 con 0x7fda0c06db50 2023-10-14T12:00:01.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.481+0000 7fda097fa700 1 -- 172.21.15.78:0/1947143157 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+12964 (secure 0 0 0) 0x7fda00057a00 con 0x7fda0c06db50 2023-10-14T12:00:01.483 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T12:00:01.483 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":35,"fsid":"4c5dc134-3092-406a-a1e1-13e6d9e926e8","created":"2023-10-14T11:57:57.363750+0000","modified":"2023-10-14T11:59:57.502053+0000","last_up_change":"2023-10-14T11:58:06.982472+0000","last_in_change":"2023-10-14T11:57:59.821373+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T11:58:09.001200+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"625a5fc0-9474-4eae-b80e-14c27a74ad19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6801","nonce":3884969780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6803","nonce":3884969780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6807","nonce":3884969780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3884969780},{"type":"v1","addr":"172.21.15.78:6805","nonce":3884969780}]},"public_addr":"172.21.15.78:6801/3884969780","cluster_addr":"172.21.15.78:6803/3884969780","heartbeat_back_addr":"172.21.15.78:6807/3884969780","heartbeat_front_addr":"172.21.15.78:6805/3884969780","state":["exists","up"]},{"osd":1,"uuid":"49d63db1-2981-49ab-9c40-c50fc744b982","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6809","nonce":2176928406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6811","nonce":2176928406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6815","nonce":2176928406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2176928406},{"type":"v1","addr":"172.21.15.78:6813","nonce":2176928406}]},"public_addr":"172.21.15.78:6809/2176928406","cluster_addr":"172.21.15.78:6811/2176928406","heartbeat_back_addr":"172.21.15.78:6815/2176928406","heartbeat_front_addr":"172.21.15.78:6813/2176928406","state":["exists","up"]},{"osd":2,"uuid":"1d7d0b29-cbd1-4717-9fe6-765a8d4998b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6825","nonce":3647044577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6827","nonce":3647044577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6831","nonce":3647044577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3647044577},{"type":"v1","addr":"172.21.15.78:6829","nonce":3647044577}]},"public_addr":"172.21.15.78:6825/3647044577","cluster_addr":"172.21.15.78:6827/3647044577","heartbeat_back_addr":"172.21.15.78:6831/3647044577","heartbeat_front_addr":"172.21.15.78:6829/3647044577","state":["exists","up"]},{"osd":3,"uuid":"2f75ded1-e9e2-40e7-99ae-45e1e425e1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6817","nonce":3584112288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6819","nonce":3584112288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6823","nonce":3584112288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3584112288},{"type":"v1","addr":"172.21.15.78:6821","nonce":3584112288}]},"public_addr":"172.21.15.78:6817/3584112288","cluster_addr":"172.21.15.78:6819/3584112288","heartbeat_back_addr":"172.21.15.78:6823/3584112288","heartbeat_front_addr":"172.21.15.78:6821/3584112288","state":["exists","up"]},{"osd":4,"uuid":"5130b432-8358-46cb-8a31-3064c976f6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6801","nonce":3732756301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6803","nonce":3732756301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6807","nonce":3732756301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3732756301},{"type":"v1","addr":"172.21.15.195:6805","nonce":3732756301}]},"public_addr":"172.21.15.195:6801/3732756301","cluster_addr":"172.21.15.195:6803/3732756301","heartbeat_back_addr":"172.21.15.195:6807/3732756301","heartbeat_front_addr":"172.21.15.195:6805/3732756301","state":["exists","up"]},{"osd":5,"uuid":"60e234a9-a506-4e40-9c19-49a58f3cf28c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6825","nonce":2857706411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6827","nonce":2857706411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6831","nonce":2857706411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2857706411},{"type":"v1","addr":"172.21.15.195:6829","nonce":2857706411}]},"public_addr":"172.21.15.195:6825/2857706411","cluster_addr":"172.21.15.195:6827/2857706411","heartbeat_back_addr":"172.21.15.195:6831/2857706411","heartbeat_front_addr":"172.21.15.195:6829/2857706411","state":["exists","up"]},{"osd":6,"uuid":"df7d2dd6-26b6-4eb1-96e5-94d1f3955b33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6810","nonce":72554374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6813","nonce":72554374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6820","nonce":72554374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":72554374},{"type":"v1","addr":"172.21.15.195:6817","nonce":72554374}]},"public_addr":"172.21.15.195:6810/72554374","cluster_addr":"172.21.15.195:6813/72554374","heartbeat_back_addr":"172.21.15.195:6820/72554374","heartbeat_front_addr":"172.21.15.195:6817/72554374","state":["exists","up"]},{"osd":7,"uuid":"47a9d4de-f56d-4aa4-9900-c4a9cf9b2f97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6809","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6811","nonce":482472979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6816","nonce":482472979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6823","nonce":482472979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":482472979},{"type":"v1","addr":"172.21.15.195:6821","nonce":482472979}]},"public_addr":"172.21.15.195:6811/482472979","cluster_addr":"172.21.15.195:6816/482472979","heartbeat_back_addr":"172.21.15.195:6823/482472979","heartbeat_front_addr":"172.21.15.195:6821/482472979","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.754003+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.005994+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.043603+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.024506+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.062585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.033767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:04.994825+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T11:58:05.038300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.195:6835/2130671508":"2023-10-15T11:59:54.495159+0000","172.21.15.195:6833/2130671508":"2023-10-15T11:59:54.495159+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T12:00:01.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.483+0000 7fda12bb7700 1 -- 172.21.15.78:0/1947143157 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd9f006af00 msgr2=0x7fd9f006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:01.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.483+0000 7fda12bb7700 1 --2- 172.21.15.78:0/1947143157 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd9f006af00 0x7fd9f006d3c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fda0c0c1170 tx=0x7fd9f8007be0 comp rx=0 tx=0).stop 2023-10-14T12:00:01.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.483+0000 7fda12bb7700 1 -- 172.21.15.78:0/1947143157 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda0c06db50 msgr2=0x7fda0c0aafc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:01.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.483+0000 7fda12bb7700 1 --2- 172.21.15.78:0/1947143157 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda0c06db50 0x7fda0c0aafc0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fda0000e810 tx=0x7fda0000ebd0 comp rx=0 tx=0).stop 2023-10-14T12:00:01.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.484+0000 7fda12bb7700 1 -- 172.21.15.78:0/1947143157 shutdown_connections 2023-10-14T12:00:01.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.484+0000 7fda12bb7700 1 --2- 172.21.15.78:0/1947143157 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fda0c154bb0 0x7fda0c1e6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:01.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.484+0000 7fda12bb7700 1 --2- 172.21.15.78:0/1947143157 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda0c06db50 0x7fda0c0aafc0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:01.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.484+0000 7fda12bb7700 1 --2- 172.21.15.78:0/1947143157 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd9f006af00 0x7fd9f006d3c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:01.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.484+0000 7fda12bb7700 1 --2- 172.21.15.78:0/1947143157 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda0c14b6b0 0x7fda0c0c6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:01.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.484+0000 7fda12bb7700 1 -- 172.21.15.78:0/1947143157 >> 172.21.15.78:0/1947143157 conn(0x7fda0c0c3ac0 msgr2=0x7fda0c0c2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:01.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.484+0000 7fda12bb7700 1 -- 172.21.15.78:0/1947143157 shutdown_connections 2023-10-14T12:00:01.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:01.484+0000 7fda12bb7700 1 -- 172.21.15.78:0/1947143157 wait complete. 2023-10-14T12:00:02.497 INFO:tasks.ceph:Scrubbing osd.0 2023-10-14T12:00:02.498 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-14T12:00:02.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.639+0000 7f6cf0036700 1 Processor -- start 2023-10-14T12:00:02.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.639+0000 7f6cf0036700 1 -- start start 2023-10-14T12:00:02.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.640+0000 7f6cf0036700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6ce806ab10 0x7f6ce815ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:02.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.640+0000 7f6cf0036700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ce8159930 con 0x7f6ce8156420 2023-10-14T12:00:02.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.640+0000 7f6cf0036700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ce8072ed0 con 0x7f6ce815a1e0 2023-10-14T12:00:02.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.640+0000 7f6cf0036700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ce8159000 con 0x7f6ce806ab10 2023-10-14T12:00:02.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.640+0000 7f6ced5d1700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6ce806ab10 0x7f6ce815ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:02.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.640+0000 7f6ceddd2700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f6ce8156420 0x7f6ce8071e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52060/0 (socket says 172.21.15.78:52060) 2023-10-14T12:00:02.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.640+0000 7f6ceddd2700 1 -- 172.21.15.78:0/2254344447 learned_addr learned my addr 172.21.15.78:0/2254344447 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T12:00:02.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.640+0000 7f6ced5d1700 1 -- 172.21.15.78:0/2254344447 >> v1:172.21.15.195:6789/0 conn(0x7f6ce815a1e0 legacy=0x7f6ce815a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T12:00:02.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.640+0000 7f6ced5d1700 1 -- 172.21.15.78:0/2254344447 >> v1:172.21.15.78:6789/0 conn(0x7f6ce8156420 legacy=0x7f6ce8071e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:02.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.640+0000 7f6ced5d1700 1 -- 172.21.15.78:0/2254344447 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce81592e0 con 0x7f6ce806ab10 2023-10-14T12:00:02.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.640+0000 7f6cecdd0700 1 -- 172.21.15.78:0/2254344447 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3577937094 0 0) 0x7f6ce8159930 con 0x7f6ce8156420 2023-10-14T12:00:02.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.640+0000 7f6ced5d1700 1 --2- 172.21.15.78:0/2254344447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ce806ab10 0x7f6ce815ccf0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f6cd8009870 tx=0x7f6cd8009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dadf80ffa3a1bb00 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:02.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.641+0000 7f6cecdd0700 1 -- 172.21.15.78:0/2254344447 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd8004230 con 0x7f6ce806ab10 2023-10-14T12:00:02.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.641+0000 7f6cecdd0700 1 -- 172.21.15.78:0/2254344447 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd8004390 con 0x7f6ce806ab10 2023-10-14T12:00:02.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.641+0000 7f6cecdd0700 1 -- 172.21.15.78:0/2254344447 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd80036a0 con 0x7f6ce806ab10 2023-10-14T12:00:02.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.641+0000 7f6cf0036700 1 -- 172.21.15.78:0/2254344447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ce806ab10 msgr2=0x7f6ce815ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:02.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.641+0000 7f6cf0036700 1 --2- 172.21.15.78:0/2254344447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ce806ab10 0x7f6ce815ccf0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f6cd8009870 tx=0x7f6cd8009b80 comp rx=0 tx=0).stop 2023-10-14T12:00:02.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.641+0000 7f6cf0036700 1 -- 172.21.15.78:0/2254344447 shutdown_connections 2023-10-14T12:00:02.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.641+0000 7f6cf0036700 1 --2- 172.21.15.78:0/2254344447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ce806ab10 0x7f6ce815ccf0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:02.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.641+0000 7f6cf0036700 1 -- 172.21.15.78:0/2254344447 >> 172.21.15.78:0/2254344447 conn(0x7f6ce80712f0 msgr2=0x7f6ce8071700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:02.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.641+0000 7f6cf0036700 1 -- 172.21.15.78:0/2254344447 shutdown_connections 2023-10-14T12:00:02.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.641+0000 7f6cf0036700 1 -- 172.21.15.78:0/2254344447 wait complete. 2023-10-14T12:00:02.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.642+0000 7f6cf0036700 1 Processor -- start 2023-10-14T12:00:02.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.642+0000 7f6cf0036700 1 -- start start 2023-10-14T12:00:02.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.643+0000 7f6cf0036700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ce806ab10 0x7f6ce81eb880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:02.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.643+0000 7f6cf0036700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ce8156420 0x7f6ce81f18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:02.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.643+0000 7f6cf0036700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ce815a1e0 0x7f6ce80a9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:02.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.643+0000 7f6cf0036700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce81ebdc0 con 0x7f6ce8156420 2023-10-14T12:00:02.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.643+0000 7f6cf0036700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce80aa640 con 0x7f6ce815a1e0 2023-10-14T12:00:02.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.643+0000 7f6cf0036700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce81f23a0 con 0x7f6ce806ab10 2023-10-14T12:00:02.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.643+0000 7f6ced5d1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ce8156420 0x7f6ce81f18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:02.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.643+0000 7f6ced5d1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ce8156420 0x7f6ce81f18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47192/0 (socket says 172.21.15.78:47192) 2023-10-14T12:00:02.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.643+0000 7f6ced5d1700 1 -- 172.21.15.78:0/1461120129 learned_addr learned my addr 172.21.15.78:0/1461120129 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:02.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.643+0000 7f6cee5d3700 1 --2- 172.21.15.78:0/1461120129 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ce815a1e0 0x7f6ce80a9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:02.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.643+0000 7f6ceddd2700 1 --2- 172.21.15.78:0/1461120129 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ce806ab10 0x7f6ce81eb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:02.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.644+0000 7f6ced5d1700 1 -- 172.21.15.78:0/1461120129 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ce806ab10 msgr2=0x7f6ce81eb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:02.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.644+0000 7f6ced5d1700 1 --2- 172.21.15.78:0/1461120129 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ce806ab10 0x7f6ce81eb880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:02.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.644+0000 7f6ced5d1700 1 -- 172.21.15.78:0/1461120129 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ce815a1e0 msgr2=0x7f6ce80a9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:02.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.644+0000 7f6ced5d1700 1 --2- 172.21.15.78:0/1461120129 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ce815a1e0 0x7f6ce80a9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:02.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.644+0000 7f6ced5d1700 1 -- 172.21.15.78:0/1461120129 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd80094a0 con 0x7f6ce8156420 2023-10-14T12:00:02.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.644+0000 7f6ceddd2700 1 --2- 172.21.15.78:0/1461120129 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ce806ab10 0x7f6ce81eb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T12:00:02.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.644+0000 7f6cee5d3700 1 --2- 172.21.15.78:0/1461120129 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ce815a1e0 0x7f6ce80a9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T12:00:02.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.644+0000 7f6ced5d1700 1 --2- 172.21.15.78:0/1461120129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ce8156420 0x7f6ce81f18f0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f6cd80090b0 tx=0x7f6cd80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:02.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.644+0000 7f6cde7fc700 1 -- 172.21.15.78:0/1461120129 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd8011830 con 0x7f6ce8156420 2023-10-14T12:00:02.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.644+0000 7f6cde7fc700 1 -- 172.21.15.78:0/1461120129 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd80119b0 con 0x7f6ce8156420 2023-10-14T12:00:02.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.644+0000 7f6cf0036700 1 -- 172.21.15.78:0/1461120129 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ce81f2680 con 0x7f6ce8156420 2023-10-14T12:00:02.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.644+0000 7f6cde7fc700 1 -- 172.21.15.78:0/1461120129 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd8011d60 con 0x7f6ce8156420 2023-10-14T12:00:02.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.644+0000 7f6cf0036700 1 -- 172.21.15.78:0/1461120129 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ce80aab70 con 0x7f6ce8156420 2023-10-14T12:00:02.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.645+0000 7f6cf0036700 1 -- 172.21.15.78:0/1461120129 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6ce8158cf0 con 0x7f6ce8156420 2023-10-14T12:00:02.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.645+0000 7f6cde7fc700 1 -- 172.21.15.78:0/1461120129 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6cd801b440 con 0x7f6ce8156420 2023-10-14T12:00:02.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.646+0000 7f6cde7fc700 1 --2- 172.21.15.78:0/1461120129 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f6ccc06af00 0x7f6ccc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:02.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.646+0000 7f6cde7fc700 1 -- 172.21.15.78:0/1461120129 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f6cd808ac80 con 0x7f6ce8156420 2023-10-14T12:00:02.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.646+0000 7f6ceddd2700 1 --2- 172.21.15.78:0/1461120129 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f6ccc06af00 0x7f6ccc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:02.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.646+0000 7f6cde7fc700 1 --2- 172.21.15.78:0/1461120129 >> [v2:172.21.15.78:6800/3884969780,v1:172.21.15.78:6801/3884969780] conn(0x7f6ccc070d60 0x7f6ccc073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:02.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.646+0000 7f6cde7fc700 1 -- 172.21.15.78:0/1461120129 --> [v2:172.21.15.78:6800/3884969780,v1:172.21.15.78:6801/3884969780] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6ccc073850 con 0x7f6ccc070d60 2023-10-14T12:00:02.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.646+0000 7f6cde7fc700 1 -- 172.21.15.78:0/1461120129 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6cd808f2a0 con 0x7f6ce8156420 2023-10-14T12:00:02.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.646+0000 7f6cee5d3700 1 --2- 172.21.15.78:0/1461120129 >> [v2:172.21.15.78:6800/3884969780,v1:172.21.15.78:6801/3884969780] conn(0x7f6ccc070d60 0x7f6ccc073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:02.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.646+0000 7f6ceddd2700 1 --2- 172.21.15.78:0/1461120129 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f6ccc06af00 0x7f6ccc06d3c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4000f60 tx=0x7f6ce4007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:02.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.646+0000 7f6cee5d3700 1 --2- 172.21.15.78:0/1461120129 >> [v2:172.21.15.78:6800/3884969780,v1:172.21.15.78:6801/3884969780] conn(0x7f6ccc070d60 0x7f6ccc073180 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:02.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.647+0000 7f6cde7fc700 1 -- 172.21.15.78:0/1461120129 <== osd.0 v2:172.21.15.78:6800/3884969780 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f6ccc073850 con 0x7f6ccc070d60 2023-10-14T12:00:02.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.662+0000 7f6cf0036700 1 -- 172.21.15.78:0/1461120129 --> [v2:172.21.15.78:6800/3884969780,v1:172.21.15.78:6801/3884969780] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f6ce81595c0 con 0x7f6ccc070d60 2023-10-14T12:00:02.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.662+0000 7f6cde7fc700 1 -- 172.21.15.78:0/1461120129 <== osd.0 v2:172.21.15.78:6800/3884969780 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f6ce0013390 con 0x7f6ccc070d60 2023-10-14T12:00:02.668 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-14T12:00:02.668 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-14T12:00:02.668 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-14T12:00:02.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.662+0000 7f6cf0036700 1 -- 172.21.15.78:0/1461120129 >> [v2:172.21.15.78:6800/3884969780,v1:172.21.15.78:6801/3884969780] conn(0x7f6ccc070d60 msgr2=0x7f6ccc073180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:02.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.662+0000 7f6cf0036700 1 --2- 172.21.15.78:0/1461120129 >> [v2:172.21.15.78:6800/3884969780,v1:172.21.15.78:6801/3884969780] conn(0x7f6ccc070d60 0x7f6ccc073180 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:02.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.662+0000 7f6cf0036700 1 -- 172.21.15.78:0/1461120129 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f6ccc06af00 msgr2=0x7f6ccc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:02.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.662+0000 7f6cf0036700 1 --2- 172.21.15.78:0/1461120129 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f6ccc06af00 0x7f6ccc06d3c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4000f60 tx=0x7f6ce4007d30 comp rx=0 tx=0).stop 2023-10-14T12:00:02.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.662+0000 7f6cf0036700 1 -- 172.21.15.78:0/1461120129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ce8156420 msgr2=0x7f6ce81f18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:02.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.662+0000 7f6cf0036700 1 --2- 172.21.15.78:0/1461120129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ce8156420 0x7f6ce81f18f0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f6cd80090b0 tx=0x7f6cd80047c0 comp rx=0 tx=0).stop 2023-10-14T12:00:02.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.662+0000 7f6cee5d3700 1 -- 172.21.15.78:0/1461120129 reap_dead start 2023-10-14T12:00:02.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.663+0000 7f6cf0036700 1 -- 172.21.15.78:0/1461120129 shutdown_connections 2023-10-14T12:00:02.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.663+0000 7f6cf0036700 1 -- 172.21.15.78:0/1461120129 >> 172.21.15.78:0/1461120129 conn(0x7f6ce80712f0 msgr2=0x7f6ce81556f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:02.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.663+0000 7f6cf0036700 1 -- 172.21.15.78:0/1461120129 shutdown_connections 2023-10-14T12:00:02.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.663+0000 7f6cf0036700 1 -- 172.21.15.78:0/1461120129 wait complete. 2023-10-14T12:00:02.676 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-14T12:00:02.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.796+0000 7f30a0390700 1 Processor -- start 2023-10-14T12:00:02.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.796+0000 7f30a0390700 1 -- start start 2023-10-14T12:00:02.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.797+0000 7f30a0390700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f309814ed80 0x7f309806a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:02.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.797+0000 7f30a0390700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f30980b9f10 0x7f30980c1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:02.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.797+0000 7f30a0390700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3098072860 con 0x7f309806e150 2023-10-14T12:00:02.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.797+0000 7f30a0390700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f309814f160 con 0x7f309814ed80 2023-10-14T12:00:02.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.797+0000 7f30a0390700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3098155760 con 0x7f30980b9f10 2023-10-14T12:00:02.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.797+0000 7f309e12c700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f309814ed80 0x7f309806a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:02.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.797+0000 7f309e92d700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f309806e150 0x7f30980c1840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52080/0 (socket says 172.21.15.78:52080) 2023-10-14T12:00:02.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.797+0000 7f309e12c700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f309814ed80 0x7f309806a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47200/0 (socket says 172.21.15.78:47200) 2023-10-14T12:00:02.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.797+0000 7f309e92d700 1 -- 172.21.15.78:0/1404627053 learned_addr learned my addr 172.21.15.78:0/1404627053 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T12:00:02.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.797+0000 7f309d92b700 1 --2- 172.21.15.78:0/1404627053 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f30980b9f10 0x7f30980c1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:02.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.797+0000 7f309e12c700 1 -- 172.21.15.78:0/1404627053 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f30980b9f10 msgr2=0x7f30980c1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T12:00:02.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.797+0000 7f309e12c700 1 --2- 172.21.15.78:0/1404627053 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f30980b9f10 0x7f30980c1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:02.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.797+0000 7f309e12c700 1 -- 172.21.15.78:0/1404627053 >> v1:172.21.15.78:6789/0 conn(0x7f309806e150 legacy=0x7f30980c1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:02.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.797+0000 7f309e12c700 1 -- 172.21.15.78:0/1404627053 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3098155a40 con 0x7f309814ed80 2023-10-14T12:00:02.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.797+0000 7f309d12a700 1 -- 172.21.15.78:0/1404627053 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3022317716 0 0) 0x7f3098072860 con 0x7f309806e150 2023-10-14T12:00:02.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.797+0000 7f309e12c700 1 --2- 172.21.15.78:0/1404627053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f309814ed80 0x7f309806a8d0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f308800bae0 tx=0x7f308800bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a05edeffe78be60b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:02.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.798+0000 7f309d12a700 1 -- 172.21.15.78:0/1404627053 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308800dd30 con 0x7f309814ed80 2023-10-14T12:00:02.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.798+0000 7f309d12a700 1 -- 172.21.15.78:0/1404627053 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f308800de90 con 0x7f309814ed80 2023-10-14T12:00:02.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.798+0000 7f309d12a700 1 -- 172.21.15.78:0/1404627053 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3088010690 con 0x7f309814ed80 2023-10-14T12:00:02.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.798+0000 7f30a0390700 1 -- 172.21.15.78:0/1404627053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f309814ed80 msgr2=0x7f309806a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:02.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.798+0000 7f30a0390700 1 --2- 172.21.15.78:0/1404627053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f309814ed80 0x7f309806a8d0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f308800bae0 tx=0x7f308800bdf0 comp rx=0 tx=0).stop 2023-10-14T12:00:02.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.798+0000 7f30a0390700 1 -- 172.21.15.78:0/1404627053 shutdown_connections 2023-10-14T12:00:02.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.798+0000 7f30a0390700 1 --2- 172.21.15.78:0/1404627053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f309814ed80 0x7f309806a8d0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:02.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.798+0000 7f30a0390700 1 --2- 172.21.15.78:0/1404627053 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f30980b9f10 0x7f30980c1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:02.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.798+0000 7f30a0390700 1 -- 172.21.15.78:0/1404627053 >> 172.21.15.78:0/1404627053 conn(0x7f30980a84a0 msgr2=0x7f30980a88b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:02.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.798+0000 7f30a0390700 1 -- 172.21.15.78:0/1404627053 shutdown_connections 2023-10-14T12:00:02.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.798+0000 7f30a0390700 1 -- 172.21.15.78:0/1404627053 wait complete. 2023-10-14T12:00:02.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.799+0000 7f30a0390700 1 Processor -- start 2023-10-14T12:00:02.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.799+0000 7f30a0390700 1 -- start start 2023-10-14T12:00:02.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.800+0000 7f30a0390700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f309806e150 0x7f309814b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:02.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.800+0000 7f30a0390700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f30980b9f10 0x7f30980b9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:02.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.800+0000 7f30a0390700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f309814ed80 0x7f30980b9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:02.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.800+0000 7f30a0390700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30980bae40 con 0x7f309806e150 2023-10-14T12:00:02.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.800+0000 7f30a0390700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309814a830 con 0x7f309814ed80 2023-10-14T12:00:02.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.800+0000 7f30a0390700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30981edaf0 con 0x7f30980b9f10 2023-10-14T12:00:02.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.800+0000 7f309e12c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f309806e150 0x7f309814b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:02.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.800+0000 7f309e12c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f309806e150 0x7f309814b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47206/0 (socket says 172.21.15.78:47206) 2023-10-14T12:00:02.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.800+0000 7f309e12c700 1 -- 172.21.15.78:0/348891523 learned_addr learned my addr 172.21.15.78:0/348891523 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:02.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.800+0000 7f309e92d700 1 --2- 172.21.15.78:0/348891523 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f309814ed80 0x7f30980b9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:02.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.800+0000 7f309d92b700 1 --2- 172.21.15.78:0/348891523 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f30980b9f10 0x7f30980b9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:02.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.800+0000 7f309e12c700 1 -- 172.21.15.78:0/348891523 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f30980b9f10 msgr2=0x7f30980b9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:02.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.800+0000 7f309e12c700 1 --2- 172.21.15.78:0/348891523 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f30980b9f10 0x7f30980b9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:02.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.800+0000 7f309e12c700 1 -- 172.21.15.78:0/348891523 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f309814ed80 msgr2=0x7f30980b9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:02.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.800+0000 7f309e12c700 1 --2- 172.21.15.78:0/348891523 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f309814ed80 0x7f30980b9850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:02.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.800+0000 7f309e12c700 1 -- 172.21.15.78:0/348891523 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f308800b740 con 0x7f309806e150 2023-10-14T12:00:02.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.800+0000 7f309e12c700 1 --2- 172.21.15.78:0/348891523 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f309806e150 0x7f309814b1c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f30980c4400 tx=0x7f3088004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:02.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.801+0000 7f308effd700 1 -- 172.21.15.78:0/348891523 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3088003da0 con 0x7f309806e150 2023-10-14T12:00:02.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.801+0000 7f308effd700 1 -- 172.21.15.78:0/348891523 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3088010470 con 0x7f309806e150 2023-10-14T12:00:02.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.801+0000 7f30a0390700 1 -- 172.21.15.78:0/348891523 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30981eaa80 con 0x7f309806e150 2023-10-14T12:00:02.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.801+0000 7f308effd700 1 -- 172.21.15.78:0/348891523 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3088010820 con 0x7f309806e150 2023-10-14T12:00:02.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.801+0000 7f30a0390700 1 -- 172.21.15.78:0/348891523 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30981ed1c0 con 0x7f309806e150 2023-10-14T12:00:02.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.801+0000 7f30a0390700 1 -- 172.21.15.78:0/348891523 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30981eddd0 con 0x7f309806e150 2023-10-14T12:00:02.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.802+0000 7f308effd700 1 -- 172.21.15.78:0/348891523 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3088023440 con 0x7f309806e150 2023-10-14T12:00:02.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.802+0000 7f308effd700 1 --2- 172.21.15.78:0/348891523 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f307c06af00 0x7f307c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:02.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.803+0000 7f309d92b700 1 --2- 172.21.15.78:0/348891523 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f307c06af00 0x7f307c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:02.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.803+0000 7f308effd700 1 -- 172.21.15.78:0/348891523 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f308808adf0 con 0x7f309806e150 2023-10-14T12:00:02.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.803+0000 7f309d92b700 1 --2- 172.21.15.78:0/348891523 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f307c06af00 0x7f307c06d3c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3098152370 tx=0x7f3094008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:02.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.807+0000 7f308effd700 1 -- 172.21.15.78:0/348891523 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3088056230 con 0x7f309806e150 2023-10-14T12:00:02.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.992+0000 7f30a0390700 1 -- 172.21.15.78:0/348891523 --> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f30980c1f50 con 0x7f307c06af00 2023-10-14T12:00:02.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.992+0000 7f308effd700 1 -- 172.21.15.78:0/348891523 <== mgr.4104 v2:172.21.15.78:6832/3171890074 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f309400b5c0 con 0x7f307c06af00 2023-10-14T12:00:02.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.994+0000 7f30a0390700 1 -- 172.21.15.78:0/348891523 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f307c06af00 msgr2=0x7f307c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:02.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.994+0000 7f30a0390700 1 --2- 172.21.15.78:0/348891523 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f307c06af00 0x7f307c06d3c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3098152370 tx=0x7f3094008040 comp rx=0 tx=0).stop 2023-10-14T12:00:02.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.994+0000 7f30a0390700 1 -- 172.21.15.78:0/348891523 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f309806e150 msgr2=0x7f309814b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:02.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.994+0000 7f30a0390700 1 --2- 172.21.15.78:0/348891523 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f309806e150 0x7f309814b1c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f30980c4400 tx=0x7f3088004750 comp rx=0 tx=0).stop 2023-10-14T12:00:02.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.994+0000 7f30a0390700 1 -- 172.21.15.78:0/348891523 shutdown_connections 2023-10-14T12:00:02.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.994+0000 7f30a0390700 1 --2- 172.21.15.78:0/348891523 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f307c06af00 0x7f307c06d3c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:02.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.994+0000 7f30a0390700 1 --2- 172.21.15.78:0/348891523 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f309814ed80 0x7f30980b9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:02.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.994+0000 7f30a0390700 1 --2- 172.21.15.78:0/348891523 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f309806e150 0x7f309814b1c0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:02.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.994+0000 7f30a0390700 1 --2- 172.21.15.78:0/348891523 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f30980b9f10 0x7f30980b9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:02.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.994+0000 7f30a0390700 1 -- 172.21.15.78:0/348891523 >> 172.21.15.78:0/348891523 conn(0x7f30980a84a0 msgr2=0x7f3098152940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:02.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.995+0000 7f30a0390700 1 -- 172.21.15.78:0/348891523 shutdown_connections 2023-10-14T12:00:03.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:02.995+0000 7f30a0390700 1 -- 172.21.15.78:0/348891523 wait complete. 2023-10-14T12:00:03.000 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 0 to deep-scrub 2023-10-14T12:00:03.008 INFO:tasks.ceph:Scrubbing osd.1 2023-10-14T12:00:03.009 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-14T12:00:03.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.130+0000 7f9399859700 1 Processor -- start 2023-10-14T12:00:03.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.130+0000 7f9399859700 1 -- start start 2023-10-14T12:00:03.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.131+0000 7f9399859700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f939414ed50 0x7f93940c6060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:03.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.131+0000 7f9399859700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f93940c5b80 0x7f93940c8100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:03.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.131+0000 7f9399859700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f93940c65a0 0x7f93940c8640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:03.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.131+0000 7f9399859700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93940c0680 con 0x7f93940c65a0 2023-10-14T12:00:03.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.131+0000 7f9399859700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93940bf440 con 0x7f93940c5b80 2023-10-14T12:00:03.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.131+0000 7f9399859700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9394155740 con 0x7f939414ed50 2023-10-14T12:00:03.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.131+0000 7f93937fe700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f93940c65a0 0x7f93940c8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:03.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.131+0000 7f93937fe700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f93940c65a0 0x7f93940c8640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47218/0 (socket says 172.21.15.78:47218) 2023-10-14T12:00:03.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.131+0000 7f93937fe700 1 -- 172.21.15.78:0/3782616817 learned_addr learned my addr 172.21.15.78:0/3782616817 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:03.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.131+0000 7f93927fc700 1 --2- 172.21.15.78:0/3782616817 >> v2:172.21.15.195:3300/0 conn(0x7f93940c5b80 0x7f93940c8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:03.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.131+0000 7f9392ffd700 1 --2- 172.21.15.78:0/3782616817 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f939414ed50 0x7f93940c6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:03.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.131+0000 7f93937fe700 1 -- 172.21.15.78:0/3782616817 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f939414ed50 msgr2=0x7f93940c6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T12:00:03.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.131+0000 7f93937fe700 1 --2- 172.21.15.78:0/3782616817 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f939414ed50 0x7f93940c6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:03.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.131+0000 7f93937fe700 1 -- 172.21.15.78:0/3782616817 >> v2:172.21.15.195:3300/0 conn(0x7f93940c5b80 msgr2=0x7f93940c8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T12:00:03.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.131+0000 7f93937fe700 1 --2- 172.21.15.78:0/3782616817 >> v2:172.21.15.195:3300/0 conn(0x7f93940c5b80 0x7f93940c8100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:03.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.131+0000 7f93937fe700 1 -- 172.21.15.78:0/3782616817 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93940bfac0 con 0x7f93940c65a0 2023-10-14T12:00:03.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.131+0000 7f93927fc700 1 --2- 172.21.15.78:0/3782616817 >> v2:172.21.15.195:3300/0 conn(0x7f93940c5b80 0x7f93940c8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T12:00:03.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.131+0000 7f9392ffd700 1 --2- 172.21.15.78:0/3782616817 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f939414ed50 0x7f93940c6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T12:00:03.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.132+0000 7f93937fe700 1 --2- 172.21.15.78:0/3782616817 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93940c65a0 0x7f93940c8640 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f93940a9c50 tx=0x7f938c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0b6ec3d2fc64316 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:03.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.132+0000 7f9391ffb700 1 -- 172.21.15.78:0/3782616817 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938c00e070 con 0x7f93940c65a0 2023-10-14T12:00:03.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.132+0000 7f9391ffb700 1 -- 172.21.15.78:0/3782616817 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f938c00b9c0 con 0x7f93940c65a0 2023-10-14T12:00:03.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.132+0000 7f9391ffb700 1 -- 172.21.15.78:0/3782616817 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938c00bd70 con 0x7f93940c65a0 2023-10-14T12:00:03.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.132+0000 7f9399859700 1 -- 172.21.15.78:0/3782616817 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93940c65a0 msgr2=0x7f93940c8640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:03.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.132+0000 7f9399859700 1 --2- 172.21.15.78:0/3782616817 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93940c65a0 0x7f93940c8640 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f93940a9c50 tx=0x7f938c00b330 comp rx=0 tx=0).stop 2023-10-14T12:00:03.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.132+0000 7f9399859700 1 -- 172.21.15.78:0/3782616817 shutdown_connections 2023-10-14T12:00:03.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.132+0000 7f9399859700 1 --2- 172.21.15.78:0/3782616817 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93940c65a0 0x7f93940c8640 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:03.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.132+0000 7f9399859700 1 --2- 172.21.15.78:0/3782616817 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f939414ed50 0x7f93940c6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:03.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.132+0000 7f9399859700 1 --2- 172.21.15.78:0/3782616817 >> v2:172.21.15.195:3300/0 conn(0x7f93940c5b80 0x7f93940c8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:03.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.132+0000 7f9399859700 1 -- 172.21.15.78:0/3782616817 >> 172.21.15.78:0/3782616817 conn(0x7f93940ca200 msgr2=0x7f93940ca5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:03.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.132+0000 7f9399859700 1 -- 172.21.15.78:0/3782616817 shutdown_connections 2023-10-14T12:00:03.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.132+0000 7f9399859700 1 -- 172.21.15.78:0/3782616817 wait complete. 2023-10-14T12:00:03.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.133+0000 7f9399859700 1 Processor -- start 2023-10-14T12:00:03.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.134+0000 7f9399859700 1 -- start start 2023-10-14T12:00:03.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.134+0000 7f9399859700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93940c5b80 0x7f93941589a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:03.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.134+0000 7f9399859700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93940c65a0 0x7f93941e1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:03.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.134+0000 7f9399859700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f939414ed50 0x7f93940c1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:03.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.134+0000 7f9399859700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9394158ee0 con 0x7f93940c65a0 2023-10-14T12:00:03.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.134+0000 7f9399859700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939414b6a0 con 0x7f93940c5b80 2023-10-14T12:00:03.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.134+0000 7f9399859700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f939414b980 con 0x7f939414ed50 2023-10-14T12:00:03.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.134+0000 7f93927fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93940c65a0 0x7f93941e1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:03.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.134+0000 7f93927fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93940c65a0 0x7f93941e1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47222/0 (socket says 172.21.15.78:47222) 2023-10-14T12:00:03.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.134+0000 7f93927fc700 1 -- 172.21.15.78:0/3564717386 learned_addr learned my addr 172.21.15.78:0/3564717386 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:03.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.135+0000 7f9392ffd700 1 --2- 172.21.15.78:0/3564717386 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93940c5b80 0x7f93941589a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:03.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.135+0000 7f93937fe700 1 --2- 172.21.15.78:0/3564717386 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f939414ed50 0x7f93940c1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:03.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.135+0000 7f93927fc700 1 -- 172.21.15.78:0/3564717386 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f939414ed50 msgr2=0x7f93940c1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:03.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.135+0000 7f93927fc700 1 --2- 172.21.15.78:0/3564717386 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f939414ed50 0x7f93940c1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:03.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.135+0000 7f93927fc700 1 -- 172.21.15.78:0/3564717386 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93940c5b80 msgr2=0x7f93941589a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:03.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.135+0000 7f93927fc700 1 --2- 172.21.15.78:0/3564717386 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93940c5b80 0x7f93941589a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:03.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.135+0000 7f93927fc700 1 -- 172.21.15.78:0/3564717386 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f938c009b30 con 0x7f93940c65a0 2023-10-14T12:00:03.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.135+0000 7f93927fc700 1 --2- 172.21.15.78:0/3564717386 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93940c65a0 0x7f93941e1900 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f938400d800 tx=0x7f938400db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:03.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.135+0000 7f937bfff700 1 -- 172.21.15.78:0/3564717386 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9384004030 con 0x7f93940c65a0 2023-10-14T12:00:03.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.135+0000 7f937bfff700 1 -- 172.21.15.78:0/3564717386 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9384004190 con 0x7f93940c65a0 2023-10-14T12:00:03.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.135+0000 7f937bfff700 1 -- 172.21.15.78:0/3564717386 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9384004520 con 0x7f93940c65a0 2023-10-14T12:00:03.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.135+0000 7f9399859700 1 -- 172.21.15.78:0/3564717386 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93940c1450 con 0x7f93940c65a0 2023-10-14T12:00:03.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.135+0000 7f9399859700 1 -- 172.21.15.78:0/3564717386 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93941e2020 con 0x7f93940c65a0 2023-10-14T12:00:03.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.136+0000 7f9399859700 1 -- 172.21.15.78:0/3564717386 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f93940c12d0 con 0x7f93940c65a0 2023-10-14T12:00:03.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.137+0000 7f937bfff700 1 -- 172.21.15.78:0/3564717386 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f93840036a0 con 0x7f93940c65a0 2023-10-14T12:00:03.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.137+0000 7f937bfff700 1 --2- 172.21.15.78:0/3564717386 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f938006af00 0x7f938006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:03.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.137+0000 7f937bfff700 1 -- 172.21.15.78:0/3564717386 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9384088650 con 0x7f93940c65a0 2023-10-14T12:00:03.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.137+0000 7f9392ffd700 1 --2- 172.21.15.78:0/3564717386 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f938006af00 0x7f938006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:03.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.137+0000 7f937bfff700 1 --2- 172.21.15.78:0/3564717386 >> [v2:172.21.15.78:6808/2176928406,v1:172.21.15.78:6809/2176928406] conn(0x7f9380070d60 0x7f9380073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:03.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.137+0000 7f937bfff700 1 -- 172.21.15.78:0/3564717386 --> [v2:172.21.15.78:6808/2176928406,v1:172.21.15.78:6809/2176928406] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9380073850 con 0x7f9380070d60 2023-10-14T12:00:03.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.137+0000 7f937bfff700 1 -- 172.21.15.78:0/3564717386 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f938408ad10 con 0x7f93940c65a0 2023-10-14T12:00:03.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.137+0000 7f93937fe700 1 --2- 172.21.15.78:0/3564717386 >> [v2:172.21.15.78:6808/2176928406,v1:172.21.15.78:6809/2176928406] conn(0x7f9380070d60 0x7f9380073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:03.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.137+0000 7f9392ffd700 1 --2- 172.21.15.78:0/3564717386 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f938006af00 0x7f938006d3c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f93880099d0 tx=0x7f9388008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:03.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.138+0000 7f93937fe700 1 --2- 172.21.15.78:0/3564717386 >> [v2:172.21.15.78:6808/2176928406,v1:172.21.15.78:6809/2176928406] conn(0x7f9380070d60 0x7f9380073180 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:03.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.138+0000 7f937bfff700 1 -- 172.21.15.78:0/3564717386 <== osd.1 v2:172.21.15.78:6808/2176928406 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f9380073850 con 0x7f9380070d60 2023-10-14T12:00:03.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.153+0000 7f9399859700 1 -- 172.21.15.78:0/3564717386 --> [v2:172.21.15.78:6808/2176928406,v1:172.21.15.78:6809/2176928406] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f9394155a20 con 0x7f9380070d60 2023-10-14T12:00:03.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.153+0000 7f937bfff700 1 -- 172.21.15.78:0/3564717386 <== osd.1 v2:172.21.15.78:6808/2176928406 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f938c0099b0 con 0x7f9380070d60 2023-10-14T12:00:03.159 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-14T12:00:03.160 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-14T12:00:03.160 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-14T12:00:03.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.153+0000 7f9399859700 1 -- 172.21.15.78:0/3564717386 >> [v2:172.21.15.78:6808/2176928406,v1:172.21.15.78:6809/2176928406] conn(0x7f9380070d60 msgr2=0x7f9380073180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:03.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.153+0000 7f9399859700 1 --2- 172.21.15.78:0/3564717386 >> [v2:172.21.15.78:6808/2176928406,v1:172.21.15.78:6809/2176928406] conn(0x7f9380070d60 0x7f9380073180 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:03.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.154+0000 7f9399859700 1 -- 172.21.15.78:0/3564717386 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f938006af00 msgr2=0x7f938006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:03.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.154+0000 7f9399859700 1 --2- 172.21.15.78:0/3564717386 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f938006af00 0x7f938006d3c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f93880099d0 tx=0x7f9388008040 comp rx=0 tx=0).stop 2023-10-14T12:00:03.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.154+0000 7f9399859700 1 -- 172.21.15.78:0/3564717386 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93940c65a0 msgr2=0x7f93941e1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:03.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.154+0000 7f9399859700 1 --2- 172.21.15.78:0/3564717386 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93940c65a0 0x7f93941e1900 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f938400d800 tx=0x7f938400db10 comp rx=0 tx=0).stop 2023-10-14T12:00:03.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.154+0000 7f93937fe700 1 -- 172.21.15.78:0/3564717386 reap_dead start 2023-10-14T12:00:03.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.154+0000 7f9399859700 1 -- 172.21.15.78:0/3564717386 shutdown_connections 2023-10-14T12:00:03.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.154+0000 7f9399859700 1 -- 172.21.15.78:0/3564717386 >> 172.21.15.78:0/3564717386 conn(0x7f93940ca200 msgr2=0x7f93940aade0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:03.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.154+0000 7f9399859700 1 -- 172.21.15.78:0/3564717386 shutdown_connections 2023-10-14T12:00:03.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.154+0000 7f9399859700 1 -- 172.21.15.78:0/3564717386 wait complete. 2023-10-14T12:00:03.167 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-14T12:00:03.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.285+0000 7f4f30f33700 1 Processor -- start 2023-10-14T12:00:03.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.285+0000 7f4f30f33700 1 -- start start 2023-10-14T12:00:03.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.286+0000 7f4f30f33700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f4f2c1533d0 0x7f4f2c0c8680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:03.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.286+0000 7f4f30f33700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4f2c0c8bc0 0x7f4f2c154960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:03.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.286+0000 7f4f30f33700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f2c1537b0 con 0x7f4f2c0a70c0 2023-10-14T12:00:03.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.286+0000 7f4f30f33700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f2c0c6fb0 con 0x7f4f2c0c8bc0 2023-10-14T12:00:03.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.286+0000 7f4f30f33700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f2c0c7f30 con 0x7f4f2c1533d0 2023-10-14T12:00:03.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.286+0000 7f4f29d9b700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4f2c0c8bc0 0x7f4f2c154960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:03.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.286+0000 7f4f29d9b700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4f2c0c8bc0 0x7f4f2c154960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47230/0 (socket says 172.21.15.78:47230) 2023-10-14T12:00:03.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.286+0000 7f4f29d9b700 1 -- 172.21.15.78:0/4169960650 learned_addr learned my addr 172.21.15.78:0/4169960650 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:03.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.286+0000 7f4f2a59c700 1 --2- 172.21.15.78:0/4169960650 >> v2:172.21.15.195:3300/0 conn(0x7f4f2c1533d0 0x7f4f2c0c8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:03.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.286+0000 7f4f29d9b700 1 -- 172.21.15.78:0/4169960650 >> v2:172.21.15.195:3300/0 conn(0x7f4f2c1533d0 msgr2=0x7f4f2c0c8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T12:00:03.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.286+0000 7f4f29d9b700 1 --2- 172.21.15.78:0/4169960650 >> v2:172.21.15.195:3300/0 conn(0x7f4f2c1533d0 0x7f4f2c0c8680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:03.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.286+0000 7f4f29d9b700 1 -- 172.21.15.78:0/4169960650 >> v1:172.21.15.195:6789/0 conn(0x7f4f2c0a70c0 legacy=0x7f4f2c154ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T12:00:03.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.286+0000 7f4f29d9b700 1 -- 172.21.15.78:0/4169960650 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f2c0c7c50 con 0x7f4f2c0c8bc0 2023-10-14T12:00:03.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.286+0000 7f4f29d9b700 1 --2- 172.21.15.78:0/4169960650 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f2c0c8bc0 0x7f4f2c154960 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c0c6e20 tx=0x7f4f20009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7db53b6193c81985 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:03.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.286+0000 7f4f2959a700 1 -- 172.21.15.78:0/4169960650 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f20004230 con 0x7f4f2c0c8bc0 2023-10-14T12:00:03.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.286+0000 7f4f2959a700 1 -- 172.21.15.78:0/4169960650 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f20004390 con 0x7f4f2c0c8bc0 2023-10-14T12:00:03.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.286+0000 7f4f2959a700 1 -- 172.21.15.78:0/4169960650 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f200036a0 con 0x7f4f2c0c8bc0 2023-10-14T12:00:03.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.287+0000 7f4f30f33700 1 -- 172.21.15.78:0/4169960650 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f2c0c8bc0 msgr2=0x7f4f2c154960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:03.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.287+0000 7f4f30f33700 1 --2- 172.21.15.78:0/4169960650 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f2c0c8bc0 0x7f4f2c154960 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c0c6e20 tx=0x7f4f20009c00 comp rx=0 tx=0).stop 2023-10-14T12:00:03.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.287+0000 7f4f30f33700 1 -- 172.21.15.78:0/4169960650 shutdown_connections 2023-10-14T12:00:03.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.287+0000 7f4f30f33700 1 --2- 172.21.15.78:0/4169960650 >> v2:172.21.15.195:3300/0 conn(0x7f4f2c1533d0 0x7f4f2c0c8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:03.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.287+0000 7f4f30f33700 1 --2- 172.21.15.78:0/4169960650 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f2c0c8bc0 0x7f4f2c154960 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:03.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.287+0000 7f4f30f33700 1 -- 172.21.15.78:0/4169960650 >> 172.21.15.78:0/4169960650 conn(0x7f4f2c0737d0 msgr2=0x7f4f2c073be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:03.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.287+0000 7f4f30f33700 1 -- 172.21.15.78:0/4169960650 shutdown_connections 2023-10-14T12:00:03.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.287+0000 7f4f30f33700 1 -- 172.21.15.78:0/4169960650 wait complete. 2023-10-14T12:00:03.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.288+0000 7f4f30f33700 1 Processor -- start 2023-10-14T12:00:03.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.288+0000 7f4f30f33700 1 -- start start 2023-10-14T12:00:03.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.289+0000 7f4f30f33700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f2c0a70c0 0x7f4f2c0c0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:03.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.289+0000 7f4f30f33700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f2c0c8bc0 0x7f4f2c06ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:03.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.289+0000 7f4f30f33700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4f2c1533d0 0x7f4f2c0ad4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:03.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.289+0000 7f4f30f33700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c0c7a70 con 0x7f4f2c0a70c0 2023-10-14T12:00:03.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.289+0000 7f4f30f33700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c0c17c0 con 0x7f4f2c0c8bc0 2023-10-14T12:00:03.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.289+0000 7f4f30f33700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c0bf7b0 con 0x7f4f2c1533d0 2023-10-14T12:00:03.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.289+0000 7f4f2a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f2c0a70c0 0x7f4f2c0c0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:03.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.289+0000 7f4f2a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f2c0a70c0 0x7f4f2c0c0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47236/0 (socket says 172.21.15.78:47236) 2023-10-14T12:00:03.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.289+0000 7f4f2a59c700 1 -- 172.21.15.78:0/1045576578 learned_addr learned my addr 172.21.15.78:0/1045576578 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:03.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.289+0000 7f4f29d9b700 1 --2- 172.21.15.78:0/1045576578 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f2c0c8bc0 0x7f4f2c06ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:03.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.289+0000 7f4f2ad9d700 1 --2- 172.21.15.78:0/1045576578 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4f2c1533d0 0x7f4f2c0ad4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:03.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.289+0000 7f4f2a59c700 1 -- 172.21.15.78:0/1045576578 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4f2c1533d0 msgr2=0x7f4f2c0ad4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:03.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.289+0000 7f4f2a59c700 1 --2- 172.21.15.78:0/1045576578 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4f2c1533d0 0x7f4f2c0ad4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:03.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.289+0000 7f4f2a59c700 1 -- 172.21.15.78:0/1045576578 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f2c0c8bc0 msgr2=0x7f4f2c06ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:03.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.289+0000 7f4f2a59c700 1 --2- 172.21.15.78:0/1045576578 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f2c0c8bc0 0x7f4f2c06ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:03.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.289+0000 7f4f2a59c700 1 -- 172.21.15.78:0/1045576578 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f20009550 con 0x7f4f2c0a70c0 2023-10-14T12:00:03.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.289+0000 7f4f29d9b700 1 --2- 172.21.15.78:0/1045576578 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f2c0c8bc0 0x7f4f2c06ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T12:00:03.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.289+0000 7f4f2ad9d700 1 --2- 172.21.15.78:0/1045576578 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4f2c1533d0 0x7f4f2c0ad4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T12:00:03.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.290+0000 7f4f2a59c700 1 --2- 172.21.15.78:0/1045576578 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f2c0a70c0 0x7f4f2c0c0630 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c00c820 tx=0x7f4f1c00cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:03.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.290+0000 7f4f1affd700 1 -- 172.21.15.78:0/1045576578 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c011800 con 0x7f4f2c0a70c0 2023-10-14T12:00:03.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.290+0000 7f4f1affd700 1 -- 172.21.15.78:0/1045576578 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f1c011960 con 0x7f4f2c0a70c0 2023-10-14T12:00:03.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.290+0000 7f4f30f33700 1 -- 172.21.15.78:0/1045576578 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f2c0c2c30 con 0x7f4f2c0a70c0 2023-10-14T12:00:03.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.290+0000 7f4f1affd700 1 -- 172.21.15.78:0/1045576578 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c011d10 con 0x7f4f2c0a70c0 2023-10-14T12:00:03.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.290+0000 7f4f30f33700 1 -- 172.21.15.78:0/1045576578 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f2c0c0f70 con 0x7f4f2c0a70c0 2023-10-14T12:00:03.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.291+0000 7f4f30f33700 1 -- 172.21.15.78:0/1045576578 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f2c0bfa20 con 0x7f4f2c0a70c0 2023-10-14T12:00:03.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.291+0000 7f4f1affd700 1 -- 172.21.15.78:0/1045576578 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4f1c018360 con 0x7f4f2c0a70c0 2023-10-14T12:00:03.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.291+0000 7f4f1affd700 1 --2- 172.21.15.78:0/1045576578 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f4f0c06af00 0x7f4f0c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:03.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.292+0000 7f4f29d9b700 1 --2- 172.21.15.78:0/1045576578 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f4f0c06af00 0x7f4f0c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:03.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.292+0000 7f4f1affd700 1 -- 172.21.15.78:0/1045576578 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4f1c089230 con 0x7f4f2c0a70c0 2023-10-14T12:00:03.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.292+0000 7f4f29d9b700 1 --2- 172.21.15.78:0/1045576578 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f4f0c06af00 0x7f4f0c06d3c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c06d990 tx=0x7f4f20008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:03.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.296+0000 7f4f1affd700 1 -- 172.21.15.78:0/1045576578 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4f1c0545e0 con 0x7f4f2c0a70c0 2023-10-14T12:00:03.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.485+0000 7f4f30f33700 1 -- 172.21.15.78:0/1045576578 --> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f2c154160 con 0x7f4f0c06af00 2023-10-14T12:00:03.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.486+0000 7f4f1affd700 1 -- 172.21.15.78:0/1045576578 <== mgr.4104 v2:172.21.15.78:6832/3171890074 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4f20020070 con 0x7f4f0c06af00 2023-10-14T12:00:03.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.488+0000 7f4f30f33700 1 -- 172.21.15.78:0/1045576578 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f4f0c06af00 msgr2=0x7f4f0c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:03.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.488+0000 7f4f30f33700 1 --2- 172.21.15.78:0/1045576578 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f4f0c06af00 0x7f4f0c06d3c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c06d990 tx=0x7f4f20008030 comp rx=0 tx=0).stop 2023-10-14T12:00:03.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.488+0000 7f4f30f33700 1 -- 172.21.15.78:0/1045576578 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f2c0a70c0 msgr2=0x7f4f2c0c0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:03.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.488+0000 7f4f30f33700 1 --2- 172.21.15.78:0/1045576578 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f2c0a70c0 0x7f4f2c0c0630 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c00c820 tx=0x7f4f1c00cbe0 comp rx=0 tx=0).stop 2023-10-14T12:00:03.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.488+0000 7f4f30f33700 1 -- 172.21.15.78:0/1045576578 shutdown_connections 2023-10-14T12:00:03.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.488+0000 7f4f30f33700 1 --2- 172.21.15.78:0/1045576578 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4f2c1533d0 0x7f4f2c0ad4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:03.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.488+0000 7f4f30f33700 1 --2- 172.21.15.78:0/1045576578 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f2c0a70c0 0x7f4f2c0c0630 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:03.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.488+0000 7f4f30f33700 1 --2- 172.21.15.78:0/1045576578 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f4f0c06af00 0x7f4f0c06d3c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:03.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.488+0000 7f4f30f33700 1 --2- 172.21.15.78:0/1045576578 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f2c0c8bc0 0x7f4f2c06ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:03.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.488+0000 7f4f30f33700 1 -- 172.21.15.78:0/1045576578 >> 172.21.15.78:0/1045576578 conn(0x7f4f2c0737d0 msgr2=0x7f4f2c154860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:03.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.488+0000 7f4f30f33700 1 -- 172.21.15.78:0/1045576578 shutdown_connections 2023-10-14T12:00:03.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.488+0000 7f4f30f33700 1 -- 172.21.15.78:0/1045576578 wait complete. 2023-10-14T12:00:03.494 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 1 to deep-scrub 2023-10-14T12:00:03.502 INFO:tasks.ceph:Scrubbing osd.2 2023-10-14T12:00:03.502 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-14T12:00:03.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.624+0000 7fb45500e700 1 Processor -- start 2023-10-14T12:00:03.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.624+0000 7fb45500e700 1 -- start start 2023-10-14T12:00:03.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.624+0000 7fb45500e700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb450152070 0x7fb450157e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:03.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.624+0000 7fb45500e700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4500acae0 con 0x7fb4501583c0 2023-10-14T12:00:03.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.624+0000 7fb45500e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4500ad010 con 0x7fb45006e410 2023-10-14T12:00:03.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.624+0000 7fb45500e700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb45006d650 con 0x7fb450152070 2023-10-14T12:00:03.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.625+0000 7fb44ed9d700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb450152070 0x7fb450157e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:03.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.625+0000 7fb44e59c700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb4501583c0 0x7fb450071e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52120/0 (socket says 172.21.15.78:52120) 2023-10-14T12:00:03.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.625+0000 7fb44ed9d700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb450152070 0x7fb450157e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47244/0 (socket says 172.21.15.78:47244) 2023-10-14T12:00:03.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.625+0000 7fb44e59c700 1 -- 172.21.15.78:0/656682445 learned_addr learned my addr 172.21.15.78:0/656682445 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T12:00:03.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.625+0000 7fb44ed9d700 1 -- 172.21.15.78:0/656682445 >> v1:172.21.15.195:6789/0 conn(0x7fb45006e410 legacy=0x7fb45015f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T12:00:03.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.625+0000 7fb44ed9d700 1 -- 172.21.15.78:0/656682445 >> v1:172.21.15.78:6789/0 conn(0x7fb4501583c0 legacy=0x7fb450071e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:03.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.625+0000 7fb44ed9d700 1 -- 172.21.15.78:0/656682445 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4500acd30 con 0x7fb450152070 2023-10-14T12:00:03.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.625+0000 7fb44dd9b700 1 -- 172.21.15.78:0/656682445 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3991165401 0 0) 0x7fb4500acae0 con 0x7fb4501583c0 2023-10-14T12:00:03.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.625+0000 7fb44ed9d700 1 --2- 172.21.15.78:0/656682445 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb450152070 0x7fb450157e80 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fb440009870 tx=0x7fb440009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68828285fe86002e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:03.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.625+0000 7fb44dd9b700 1 -- 172.21.15.78:0/656682445 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb440004230 con 0x7fb450152070 2023-10-14T12:00:03.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.625+0000 7fb44dd9b700 1 -- 172.21.15.78:0/656682445 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb440004390 con 0x7fb450152070 2023-10-14T12:00:03.633 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.625+0000 7fb44dd9b700 1 -- 172.21.15.78:0/656682445 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4400036a0 con 0x7fb450152070 2023-10-14T12:00:03.633 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.626+0000 7fb45500e700 1 -- 172.21.15.78:0/656682445 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb450152070 msgr2=0x7fb450157e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:03.633 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.626+0000 7fb45500e700 1 --2- 172.21.15.78:0/656682445 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb450152070 0x7fb450157e80 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fb440009870 tx=0x7fb440009b80 comp rx=0 tx=0).stop 2023-10-14T12:00:03.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.626+0000 7fb45500e700 1 -- 172.21.15.78:0/656682445 shutdown_connections 2023-10-14T12:00:03.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.626+0000 7fb45500e700 1 --2- 172.21.15.78:0/656682445 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb450152070 0x7fb450157e80 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:03.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.626+0000 7fb45500e700 1 -- 172.21.15.78:0/656682445 >> 172.21.15.78:0/656682445 conn(0x7fb4500712f0 msgr2=0x7fb450071700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:03.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.626+0000 7fb45500e700 1 -- 172.21.15.78:0/656682445 shutdown_connections 2023-10-14T12:00:03.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.626+0000 7fb45500e700 1 -- 172.21.15.78:0/656682445 wait complete. 2023-10-14T12:00:03.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.627+0000 7fb45500e700 1 Processor -- start 2023-10-14T12:00:03.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.627+0000 7fb45500e700 1 -- start start 2023-10-14T12:00:03.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.628+0000 7fb45500e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb45006e410 0x7fb4501edf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:03.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.628+0000 7fb45500e700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb450152070 0x7fb4501506b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:03.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.628+0000 7fb45500e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb4501583c0 0x7fb45014fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:03.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.628+0000 7fb45500e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4501edaf0 con 0x7fb45006e410 2023-10-14T12:00:03.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.628+0000 7fb45500e700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb450153440 con 0x7fb4501583c0 2023-10-14T12:00:03.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.628+0000 7fb45500e700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4501ee4c0 con 0x7fb450152070 2023-10-14T12:00:03.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.628+0000 7fb44ed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb45006e410 0x7fb4501edf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:03.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.628+0000 7fb44ed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb45006e410 0x7fb4501edf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47250/0 (socket says 172.21.15.78:47250) 2023-10-14T12:00:03.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.628+0000 7fb44ed9d700 1 -- 172.21.15.78:0/3459978407 learned_addr learned my addr 172.21.15.78:0/3459978407 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:03.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.628+0000 7fb44f59e700 1 --2- 172.21.15.78:0/3459978407 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb4501583c0 0x7fb45014fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:03.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.628+0000 7fb44e59c700 1 --2- 172.21.15.78:0/3459978407 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb450152070 0x7fb4501506b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:03.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.628+0000 7fb44ed9d700 1 -- 172.21.15.78:0/3459978407 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb450152070 msgr2=0x7fb4501506b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:03.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.628+0000 7fb44ed9d700 1 --2- 172.21.15.78:0/3459978407 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb450152070 0x7fb4501506b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:03.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.628+0000 7fb44ed9d700 1 -- 172.21.15.78:0/3459978407 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb4501583c0 msgr2=0x7fb45014fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:03.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.628+0000 7fb44ed9d700 1 --2- 172.21.15.78:0/3459978407 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb4501583c0 0x7fb45014fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:03.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.628+0000 7fb44ed9d700 1 -- 172.21.15.78:0/3459978407 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4400094a0 con 0x7fb45006e410 2023-10-14T12:00:03.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.629+0000 7fb44ed9d700 1 --2- 172.21.15.78:0/3459978407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb45006e410 0x7fb4501edf80 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fb440000c00 tx=0x7fb440004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:03.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.629+0000 7fb4377fe700 1 -- 172.21.15.78:0/3459978407 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb440011830 con 0x7fb45006e410 2023-10-14T12:00:03.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.629+0000 7fb4377fe700 1 -- 172.21.15.78:0/3459978407 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb44001e070 con 0x7fb45006e410 2023-10-14T12:00:03.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.629+0000 7fb4377fe700 1 -- 172.21.15.78:0/3459978407 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb440011bf0 con 0x7fb45006e410 2023-10-14T12:00:03.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.629+0000 7fb45500e700 1 -- 172.21.15.78:0/3459978407 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb450150bf0 con 0x7fb45006e410 2023-10-14T12:00:03.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.629+0000 7fb45500e700 1 -- 172.21.15.78:0/3459978407 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb450153160 con 0x7fb45006e410 2023-10-14T12:00:03.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.630+0000 7fb45500e700 1 -- 172.21.15.78:0/3459978407 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb4500c1180 con 0x7fb45006e410 2023-10-14T12:00:03.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.630+0000 7fb4377fe700 1 -- 172.21.15.78:0/3459978407 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb440011d50 con 0x7fb45006e410 2023-10-14T12:00:03.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.631+0000 7fb4377fe700 1 --2- 172.21.15.78:0/3459978407 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb43c06af00 0x7fb43c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:03.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.631+0000 7fb4377fe700 1 -- 172.21.15.78:0/3459978407 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb44008a610 con 0x7fb45006e410 2023-10-14T12:00:03.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.631+0000 7fb44e59c700 1 --2- 172.21.15.78:0/3459978407 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb43c06af00 0x7fb43c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:03.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.631+0000 7fb4377fe700 1 --2- 172.21.15.78:0/3459978407 >> [v2:172.21.15.78:6824/3647044577,v1:172.21.15.78:6825/3647044577] conn(0x7fb43c070d60 0x7fb43c073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:03.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.631+0000 7fb4377fe700 1 -- 172.21.15.78:0/3459978407 --> [v2:172.21.15.78:6824/3647044577,v1:172.21.15.78:6825/3647044577] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb43c073850 con 0x7fb43c070d60 2023-10-14T12:00:03.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.631+0000 7fb4377fe700 1 -- 172.21.15.78:0/3459978407 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb44008cbf0 con 0x7fb45006e410 2023-10-14T12:00:03.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.631+0000 7fb44f59e700 1 --2- 172.21.15.78:0/3459978407 >> [v2:172.21.15.78:6824/3647044577,v1:172.21.15.78:6825/3647044577] conn(0x7fb43c070d60 0x7fb43c073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:03.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.631+0000 7fb44e59c700 1 --2- 172.21.15.78:0/3459978407 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb43c06af00 0x7fb43c06d3c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb444000f60 tx=0x7fb444007c90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:03.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.631+0000 7fb44f59e700 1 --2- 172.21.15.78:0/3459978407 >> [v2:172.21.15.78:6824/3647044577,v1:172.21.15.78:6825/3647044577] conn(0x7fb43c070d60 0x7fb43c073180 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:03.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.632+0000 7fb4377fe700 1 -- 172.21.15.78:0/3459978407 <== osd.2 v2:172.21.15.78:6824/3647044577 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fb43c073850 con 0x7fb43c070d60 2023-10-14T12:00:03.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.646+0000 7fb45500e700 1 -- 172.21.15.78:0/3459978407 --> [v2:172.21.15.78:6824/3647044577,v1:172.21.15.78:6825/3647044577] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb4501edc30 con 0x7fb43c070d60 2023-10-14T12:00:03.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.647+0000 7fb4377fe700 1 -- 172.21.15.78:0/3459978407 <== osd.2 v2:172.21.15.78:6824/3647044577 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fb448013390 con 0x7fb43c070d60 2023-10-14T12:00:03.650 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-14T12:00:03.650 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-14T12:00:03.651 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-14T12:00:03.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.647+0000 7fb45500e700 1 -- 172.21.15.78:0/3459978407 >> [v2:172.21.15.78:6824/3647044577,v1:172.21.15.78:6825/3647044577] conn(0x7fb43c070d60 msgr2=0x7fb43c073180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:03.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.647+0000 7fb45500e700 1 --2- 172.21.15.78:0/3459978407 >> [v2:172.21.15.78:6824/3647044577,v1:172.21.15.78:6825/3647044577] conn(0x7fb43c070d60 0x7fb43c073180 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:03.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.647+0000 7fb45500e700 1 -- 172.21.15.78:0/3459978407 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb43c06af00 msgr2=0x7fb43c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:03.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.647+0000 7fb45500e700 1 --2- 172.21.15.78:0/3459978407 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb43c06af00 0x7fb43c06d3c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb444000f60 tx=0x7fb444007c90 comp rx=0 tx=0).stop 2023-10-14T12:00:03.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.647+0000 7fb45500e700 1 -- 172.21.15.78:0/3459978407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb45006e410 msgr2=0x7fb4501edf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:03.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.647+0000 7fb45500e700 1 --2- 172.21.15.78:0/3459978407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb45006e410 0x7fb4501edf80 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fb440000c00 tx=0x7fb440004750 comp rx=0 tx=0).stop 2023-10-14T12:00:03.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.647+0000 7fb44f59e700 1 -- 172.21.15.78:0/3459978407 reap_dead start 2023-10-14T12:00:03.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.647+0000 7fb45500e700 1 -- 172.21.15.78:0/3459978407 shutdown_connections 2023-10-14T12:00:03.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.647+0000 7fb45500e700 1 -- 172.21.15.78:0/3459978407 >> 172.21.15.78:0/3459978407 conn(0x7fb4500712f0 msgr2=0x7fb45014d040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:03.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.647+0000 7fb45500e700 1 -- 172.21.15.78:0/3459978407 shutdown_connections 2023-10-14T12:00:03.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.647+0000 7fb45500e700 1 -- 172.21.15.78:0/3459978407 wait complete. 2023-10-14T12:00:03.661 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-14T12:00:03.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.779+0000 7fd363ef3700 1 Processor -- start 2023-10-14T12:00:03.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.779+0000 7fd363ef3700 1 -- start start 2023-10-14T12:00:03.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.780+0000 7fd363ef3700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd35c14ecc0 0x7fd35c151730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:03.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.780+0000 7fd363ef3700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd35c0c1c00 0x7fd35c0c2050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:03.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.780+0000 7fd363ef3700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35c06e080 con 0x7fd35c06da10 2023-10-14T12:00:03.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.780+0000 7fd363ef3700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd35c06de70 con 0x7fd35c0c1c00 2023-10-14T12:00:03.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.780+0000 7fd363ef3700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd35c06e360 con 0x7fd35c14ecc0 2023-10-14T12:00:03.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.780+0000 7fd362490700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd35c0c1c00 0x7fd35c0c2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:03.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.780+0000 7fd362490700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd35c0c1c00 0x7fd35c0c2050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47264/0 (socket says 172.21.15.78:47264) 2023-10-14T12:00:03.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.780+0000 7fd362490700 1 -- 172.21.15.78:0/1034715250 learned_addr learned my addr 172.21.15.78:0/1034715250 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:03.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.781+0000 7fd361c8f700 1 --2- 172.21.15.78:0/1034715250 >> v2:172.21.15.195:3300/0 conn(0x7fd35c14ecc0 0x7fd35c151730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:03.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.781+0000 7fd362490700 1 -- 172.21.15.78:0/1034715250 >> v2:172.21.15.195:3300/0 conn(0x7fd35c14ecc0 msgr2=0x7fd35c151730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T12:00:03.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.781+0000 7fd362490700 1 --2- 172.21.15.78:0/1034715250 >> v2:172.21.15.195:3300/0 conn(0x7fd35c14ecc0 0x7fd35c151730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:03.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.781+0000 7fd362490700 1 -- 172.21.15.78:0/1034715250 >> v1:172.21.15.195:6789/0 conn(0x7fd35c06da10 legacy=0x7fd35c06eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T12:00:03.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.781+0000 7fd362490700 1 -- 172.21.15.78:0/1034715250 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd35c06e640 con 0x7fd35c0c1c00 2023-10-14T12:00:03.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.781+0000 7fd362490700 1 --2- 172.21.15.78:0/1034715250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd35c0c1c00 0x7fd35c0c2050 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd35c06c8b0 tx=0x7fd35400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8700a72e5bf4cbe server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:03.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.781+0000 7fd360c8d700 1 -- 172.21.15.78:0/1034715250 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35400e070 con 0x7fd35c0c1c00 2023-10-14T12:00:03.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.781+0000 7fd360c8d700 1 -- 172.21.15.78:0/1034715250 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd354003a70 con 0x7fd35c0c1c00 2023-10-14T12:00:03.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.781+0000 7fd360c8d700 1 -- 172.21.15.78:0/1034715250 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd354003e20 con 0x7fd35c0c1c00 2023-10-14T12:00:03.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.781+0000 7fd363ef3700 1 -- 172.21.15.78:0/1034715250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd35c0c1c00 msgr2=0x7fd35c0c2050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:03.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.781+0000 7fd363ef3700 1 --2- 172.21.15.78:0/1034715250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd35c0c1c00 0x7fd35c0c2050 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd35c06c8b0 tx=0x7fd35400b330 comp rx=0 tx=0).stop 2023-10-14T12:00:03.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.781+0000 7fd363ef3700 1 -- 172.21.15.78:0/1034715250 shutdown_connections 2023-10-14T12:00:03.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.781+0000 7fd363ef3700 1 --2- 172.21.15.78:0/1034715250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd35c0c1c00 0x7fd35c0c2050 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:03.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.781+0000 7fd363ef3700 1 --2- 172.21.15.78:0/1034715250 >> v2:172.21.15.195:3300/0 conn(0x7fd35c14ecc0 0x7fd35c151730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:03.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.781+0000 7fd363ef3700 1 -- 172.21.15.78:0/1034715250 >> 172.21.15.78:0/1034715250 conn(0x7fd35c0c3a20 msgr2=0x7fd35c0c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:03.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.782+0000 7fd363ef3700 1 -- 172.21.15.78:0/1034715250 shutdown_connections 2023-10-14T12:00:03.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.782+0000 7fd363ef3700 1 -- 172.21.15.78:0/1034715250 wait complete. 2023-10-14T12:00:03.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.783+0000 7fd363ef3700 1 Processor -- start 2023-10-14T12:00:03.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.783+0000 7fd363ef3700 1 -- start start 2023-10-14T12:00:03.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.783+0000 7fd363ef3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd35c06da10 0x7fd35c06ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:03.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.783+0000 7fd363ef3700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd35c0c1c00 0x7fd35c140740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:03.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.783+0000 7fd363ef3700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd35c14ecc0 0x7fd35c0bd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:03.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.783+0000 7fd363ef3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c0be530 con 0x7fd35c06da10 2023-10-14T12:00:03.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.783+0000 7fd363ef3700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c1e7c50 con 0x7fd35c14ecc0 2023-10-14T12:00:03.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.783+0000 7fd363ef3700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c1e8be0 con 0x7fd35c0c1c00 2023-10-14T12:00:03.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.783+0000 7fd361c8f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd35c06da10 0x7fd35c06ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:03.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.783+0000 7fd361c8f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd35c06da10 0x7fd35c06ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47266/0 (socket says 172.21.15.78:47266) 2023-10-14T12:00:03.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.783+0000 7fd361c8f700 1 -- 172.21.15.78:0/2792396908 learned_addr learned my addr 172.21.15.78:0/2792396908 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:03.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.784+0000 7fd362490700 1 --2- 172.21.15.78:0/2792396908 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd35c14ecc0 0x7fd35c0bd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:03.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.784+0000 7fd36148e700 1 --2- 172.21.15.78:0/2792396908 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd35c0c1c00 0x7fd35c140740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:03.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.784+0000 7fd361c8f700 1 -- 172.21.15.78:0/2792396908 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd35c0c1c00 msgr2=0x7fd35c140740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:03.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.784+0000 7fd361c8f700 1 --2- 172.21.15.78:0/2792396908 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd35c0c1c00 0x7fd35c140740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:03.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.784+0000 7fd361c8f700 1 -- 172.21.15.78:0/2792396908 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd35c14ecc0 msgr2=0x7fd35c0bd4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:03.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.784+0000 7fd361c8f700 1 --2- 172.21.15.78:0/2792396908 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd35c14ecc0 0x7fd35c0bd4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:03.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.784+0000 7fd361c8f700 1 -- 172.21.15.78:0/2792396908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd354009b30 con 0x7fd35c06da10 2023-10-14T12:00:03.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.784+0000 7fd362490700 1 --2- 172.21.15.78:0/2792396908 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd35c14ecc0 0x7fd35c0bd4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T12:00:03.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.784+0000 7fd361c8f700 1 --2- 172.21.15.78:0/2792396908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd35c06da10 0x7fd35c06ab40 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fd34c00c820 tx=0x7fd34c00cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:03.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.784+0000 7fd3527fc700 1 -- 172.21.15.78:0/2792396908 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34c011800 con 0x7fd35c06da10 2023-10-14T12:00:03.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.784+0000 7fd3527fc700 1 -- 172.21.15.78:0/2792396908 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd34c011960 con 0x7fd35c06da10 2023-10-14T12:00:03.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.784+0000 7fd363ef3700 1 -- 172.21.15.78:0/2792396908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd35c1e7e60 con 0x7fd35c06da10 2023-10-14T12:00:03.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.784+0000 7fd3527fc700 1 -- 172.21.15.78:0/2792396908 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34c011cf0 con 0x7fd35c06da10 2023-10-14T12:00:03.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.784+0000 7fd363ef3700 1 -- 172.21.15.78:0/2792396908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd35c1e83e0 con 0x7fd35c06da10 2023-10-14T12:00:03.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.785+0000 7fd363ef3700 1 -- 172.21.15.78:0/2792396908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd35c074b80 con 0x7fd35c06da10 2023-10-14T12:00:03.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.785+0000 7fd3527fc700 1 -- 172.21.15.78:0/2792396908 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd34c018360 con 0x7fd35c06da10 2023-10-14T12:00:03.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.786+0000 7fd3527fc700 1 --2- 172.21.15.78:0/2792396908 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd34006af00 0x7fd34006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:03.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.786+0000 7fd3527fc700 1 -- 172.21.15.78:0/2792396908 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd34c089130 con 0x7fd35c06da10 2023-10-14T12:00:03.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.786+0000 7fd36148e700 1 --2- 172.21.15.78:0/2792396908 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd34006af00 0x7fd34006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:03.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.786+0000 7fd36148e700 1 --2- 172.21.15.78:0/2792396908 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd34006af00 0x7fd34006d3c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd35c0728e0 tx=0x7fd358009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:03.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.790+0000 7fd3527fc700 1 -- 172.21.15.78:0/2792396908 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd34c054570 con 0x7fd35c06da10 2023-10-14T12:00:03.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.979+0000 7fd363ef3700 1 -- 172.21.15.78:0/2792396908 --> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fd35c141150 con 0x7fd34006af00 2023-10-14T12:00:03.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.979+0000 7fd3527fc700 1 -- 172.21.15.78:0/2792396908 <== mgr.4104 v2:172.21.15.78:6832/3171890074 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd3580050d0 con 0x7fd34006af00 2023-10-14T12:00:03.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.981+0000 7fd363ef3700 1 -- 172.21.15.78:0/2792396908 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd34006af00 msgr2=0x7fd34006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:03.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.981+0000 7fd363ef3700 1 --2- 172.21.15.78:0/2792396908 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd34006af00 0x7fd34006d3c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd35c0728e0 tx=0x7fd358009040 comp rx=0 tx=0).stop 2023-10-14T12:00:03.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.981+0000 7fd363ef3700 1 -- 172.21.15.78:0/2792396908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd35c06da10 msgr2=0x7fd35c06ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:03.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.981+0000 7fd363ef3700 1 --2- 172.21.15.78:0/2792396908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd35c06da10 0x7fd35c06ab40 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fd34c00c820 tx=0x7fd34c00cbe0 comp rx=0 tx=0).stop 2023-10-14T12:00:03.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.981+0000 7fd363ef3700 1 -- 172.21.15.78:0/2792396908 shutdown_connections 2023-10-14T12:00:03.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.981+0000 7fd363ef3700 1 --2- 172.21.15.78:0/2792396908 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fd34006af00 0x7fd34006d3c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:03.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.981+0000 7fd363ef3700 1 --2- 172.21.15.78:0/2792396908 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd35c14ecc0 0x7fd35c0bd4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:03.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.981+0000 7fd363ef3700 1 --2- 172.21.15.78:0/2792396908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd35c06da10 0x7fd35c06ab40 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:03.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.981+0000 7fd363ef3700 1 --2- 172.21.15.78:0/2792396908 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd35c0c1c00 0x7fd35c140740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:03.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.981+0000 7fd363ef3700 1 -- 172.21.15.78:0/2792396908 >> 172.21.15.78:0/2792396908 conn(0x7fd35c0c3a20 msgr2=0x7fd35c150d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:03.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.981+0000 7fd363ef3700 1 -- 172.21.15.78:0/2792396908 shutdown_connections 2023-10-14T12:00:03.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:03.981+0000 7fd363ef3700 1 -- 172.21.15.78:0/2792396908 wait complete. 2023-10-14T12:00:03.987 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 2 to deep-scrub 2023-10-14T12:00:03.995 INFO:tasks.ceph:Scrubbing osd.3 2023-10-14T12:00:03.995 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-10-14T12:00:04.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.117+0000 7ffb6f64f700 1 Processor -- start 2023-10-14T12:00:04.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.117+0000 7ffb6f64f700 1 -- start start 2023-10-14T12:00:04.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.118+0000 7ffb6f64f700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ffb68152070 0x7ffb68157e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:04.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.118+0000 7ffb6f64f700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ffb681583c0 0x7ffb68071e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:04.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.118+0000 7ffb6f64f700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffb6806e410 0x7ffb680c2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:04.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.118+0000 7ffb6f64f700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb680acae0 con 0x7ffb68152070 2023-10-14T12:00:04.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.118+0000 7ffb6f64f700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb680ad010 con 0x7ffb681583c0 2023-10-14T12:00:04.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.118+0000 7ffb6f64f700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6806d650 con 0x7ffb6806e410 2023-10-14T12:00:04.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.118+0000 7ffb6d3eb700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ffb68152070 0x7ffb68157e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:04.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.118+0000 7ffb6d3eb700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ffb68152070 0x7ffb68157e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47274/0 (socket says 172.21.15.78:47274) 2023-10-14T12:00:04.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.118+0000 7ffb6d3eb700 1 -- 172.21.15.78:0/2114569398 learned_addr learned my addr 172.21.15.78:0/2114569398 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:04.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.119+0000 7ffb6cbea700 1 --2- 172.21.15.78:0/2114569398 >> v2:172.21.15.195:3300/0 conn(0x7ffb681583c0 0x7ffb68071e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:04.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.119+0000 7ffb6dbec700 1 --2- 172.21.15.78:0/2114569398 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffb6806e410 0x7ffb680c2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:04.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.119+0000 7ffb6d3eb700 1 -- 172.21.15.78:0/2114569398 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffb6806e410 msgr2=0x7ffb680c2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T12:00:04.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.119+0000 7ffb6d3eb700 1 --2- 172.21.15.78:0/2114569398 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffb6806e410 0x7ffb680c2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.119+0000 7ffb6d3eb700 1 -- 172.21.15.78:0/2114569398 >> v2:172.21.15.195:3300/0 conn(0x7ffb681583c0 msgr2=0x7ffb68071e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T12:00:04.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.119+0000 7ffb6d3eb700 1 --2- 172.21.15.78:0/2114569398 >> v2:172.21.15.195:3300/0 conn(0x7ffb681583c0 0x7ffb68071e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.119+0000 7ffb6d3eb700 1 -- 172.21.15.78:0/2114569398 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb680acd30 con 0x7ffb68152070 2023-10-14T12:00:04.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.119+0000 7ffb6cbea700 1 --2- 172.21.15.78:0/2114569398 >> v2:172.21.15.195:3300/0 conn(0x7ffb681583c0 0x7ffb68071e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T12:00:04.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.119+0000 7ffb6d3eb700 1 --2- 172.21.15.78:0/2114569398 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb68152070 0x7ffb68157e80 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ffb68074680 tx=0x7ffb58009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91f60d2a19dead67 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:04.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.119+0000 7ffb5ffff700 1 -- 172.21.15.78:0/2114569398 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5800b870 con 0x7ffb68152070 2023-10-14T12:00:04.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.119+0000 7ffb5ffff700 1 -- 172.21.15.78:0/2114569398 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb5800b9d0 con 0x7ffb68152070 2023-10-14T12:00:04.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.119+0000 7ffb5ffff700 1 -- 172.21.15.78:0/2114569398 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5800bd60 con 0x7ffb68152070 2023-10-14T12:00:04.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.120+0000 7ffb6f64f700 1 -- 172.21.15.78:0/2114569398 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb68152070 msgr2=0x7ffb68157e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:04.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.120+0000 7ffb6f64f700 1 --2- 172.21.15.78:0/2114569398 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb68152070 0x7ffb68157e80 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ffb68074680 tx=0x7ffb58009bd0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.120+0000 7ffb6f64f700 1 -- 172.21.15.78:0/2114569398 shutdown_connections 2023-10-14T12:00:04.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.120+0000 7ffb6f64f700 1 --2- 172.21.15.78:0/2114569398 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffb6806e410 0x7ffb680c2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.120+0000 7ffb6f64f700 1 --2- 172.21.15.78:0/2114569398 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb68152070 0x7ffb68157e80 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.120+0000 7ffb6f64f700 1 --2- 172.21.15.78:0/2114569398 >> v2:172.21.15.195:3300/0 conn(0x7ffb681583c0 0x7ffb68071e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.120+0000 7ffb6f64f700 1 -- 172.21.15.78:0/2114569398 >> 172.21.15.78:0/2114569398 conn(0x7ffb680712f0 msgr2=0x7ffb68071700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:04.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.120+0000 7ffb6f64f700 1 -- 172.21.15.78:0/2114569398 shutdown_connections 2023-10-14T12:00:04.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.120+0000 7ffb6f64f700 1 -- 172.21.15.78:0/2114569398 wait complete. 2023-10-14T12:00:04.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.121+0000 7ffb6f64f700 1 Processor -- start 2023-10-14T12:00:04.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.121+0000 7ffb6f64f700 1 -- start start 2023-10-14T12:00:04.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.122+0000 7ffb6f64f700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffb6806e410 0x7ffb680c6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:04.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.122+0000 7ffb6f64f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb68152070 0x7ffb68151130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:04.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.122+0000 7ffb6f64f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffb681583c0 0x7ffb68147b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:04.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.122+0000 7ffb6f64f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb681e9b50 con 0x7ffb68152070 2023-10-14T12:00:04.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.122+0000 7ffb6f64f700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb681e9e10 con 0x7ffb681583c0 2023-10-14T12:00:04.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.122+0000 7ffb6f64f700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb681ea0f0 con 0x7ffb6806e410 2023-10-14T12:00:04.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.122+0000 7ffb6cbea700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb68152070 0x7ffb68151130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:04.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.122+0000 7ffb6cbea700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb68152070 0x7ffb68151130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47282/0 (socket says 172.21.15.78:47282) 2023-10-14T12:00:04.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.122+0000 7ffb6cbea700 1 -- 172.21.15.78:0/1489747491 learned_addr learned my addr 172.21.15.78:0/1489747491 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:04.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.122+0000 7ffb6dbec700 1 --2- 172.21.15.78:0/1489747491 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffb681583c0 0x7ffb68147b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:04.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.122+0000 7ffb6d3eb700 1 --2- 172.21.15.78:0/1489747491 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffb6806e410 0x7ffb680c6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:04.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.122+0000 7ffb6cbea700 1 -- 172.21.15.78:0/1489747491 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffb6806e410 msgr2=0x7ffb680c6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:04.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.122+0000 7ffb6cbea700 1 --2- 172.21.15.78:0/1489747491 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffb6806e410 0x7ffb680c6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.122+0000 7ffb6cbea700 1 -- 172.21.15.78:0/1489747491 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffb681583c0 msgr2=0x7ffb68147b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:04.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.122+0000 7ffb6cbea700 1 --2- 172.21.15.78:0/1489747491 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffb681583c0 0x7ffb68147b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.122+0000 7ffb6cbea700 1 -- 172.21.15.78:0/1489747491 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb58009550 con 0x7ffb68152070 2023-10-14T12:00:04.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.122+0000 7ffb6d3eb700 1 --2- 172.21.15.78:0/1489747491 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffb6806e410 0x7ffb680c6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T12:00:04.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.122+0000 7ffb6cbea700 1 --2- 172.21.15.78:0/1489747491 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb68152070 0x7ffb68151130 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7ffb6400c7f0 tx=0x7ffb6400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:04.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.122+0000 7ffb5dffb700 1 -- 172.21.15.78:0/1489747491 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb64011800 con 0x7ffb68152070 2023-10-14T12:00:04.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.122+0000 7ffb5dffb700 1 -- 172.21.15.78:0/1489747491 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb64011960 con 0x7ffb68152070 2023-10-14T12:00:04.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.122+0000 7ffb5dffb700 1 -- 172.21.15.78:0/1489747491 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb64011d10 con 0x7ffb68152070 2023-10-14T12:00:04.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.122+0000 7ffb6f64f700 1 -- 172.21.15.78:0/1489747491 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb681ea3d0 con 0x7ffb68152070 2023-10-14T12:00:04.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.123+0000 7ffb6f64f700 1 -- 172.21.15.78:0/1489747491 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb680c6aa0 con 0x7ffb68152070 2023-10-14T12:00:04.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.123+0000 7ffb6f64f700 1 -- 172.21.15.78:0/1489747491 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ffb680c1180 con 0x7ffb68152070 2023-10-14T12:00:04.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.124+0000 7ffb5dffb700 1 -- 172.21.15.78:0/1489747491 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ffb64018360 con 0x7ffb68152070 2023-10-14T12:00:04.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.124+0000 7ffb5dffb700 1 --2- 172.21.15.78:0/1489747491 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ffb4c06af00 0x7ffb4c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:04.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.124+0000 7ffb5dffb700 1 -- 172.21.15.78:0/1489747491 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ffb64089220 con 0x7ffb68152070 2023-10-14T12:00:04.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.124+0000 7ffb6d3eb700 1 --2- 172.21.15.78:0/1489747491 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ffb4c06af00 0x7ffb4c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:04.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.124+0000 7ffb5dffb700 1 --2- 172.21.15.78:0/1489747491 >> [v2:172.21.15.78:6816/3584112288,v1:172.21.15.78:6817/3584112288] conn(0x7ffb4c070d60 0x7ffb4c073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:04.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.124+0000 7ffb5dffb700 1 -- 172.21.15.78:0/1489747491 --> [v2:172.21.15.78:6816/3584112288,v1:172.21.15.78:6817/3584112288] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffb4c073850 con 0x7ffb4c070d60 2023-10-14T12:00:04.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.124+0000 7ffb5dffb700 1 -- 172.21.15.78:0/1489747491 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb6408b800 con 0x7ffb68152070 2023-10-14T12:00:04.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.125+0000 7ffb6dbec700 1 --2- 172.21.15.78:0/1489747491 >> [v2:172.21.15.78:6816/3584112288,v1:172.21.15.78:6817/3584112288] conn(0x7ffb4c070d60 0x7ffb4c073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:04.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.125+0000 7ffb6d3eb700 1 --2- 172.21.15.78:0/1489747491 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ffb4c06af00 0x7ffb4c06d3c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ffb58000c00 tx=0x7ffb58008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:04.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.125+0000 7ffb6dbec700 1 --2- 172.21.15.78:0/1489747491 >> [v2:172.21.15.78:6816/3584112288,v1:172.21.15.78:6817/3584112288] conn(0x7ffb4c070d60 0x7ffb4c073180 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:04.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.126+0000 7ffb5dffb700 1 -- 172.21.15.78:0/1489747491 <== osd.3 v2:172.21.15.78:6816/3584112288 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7ffb4c073850 con 0x7ffb4c070d60 2023-10-14T12:00:04.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.140+0000 7ffb6f64f700 1 -- 172.21.15.78:0/1489747491 --> [v2:172.21.15.78:6816/3584112288,v1:172.21.15.78:6817/3584112288] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ffb68074b80 con 0x7ffb4c070d60 2023-10-14T12:00:04.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.141+0000 7ffb5dffb700 1 -- 172.21.15.78:0/1489747491 <== osd.3 v2:172.21.15.78:6816/3584112288 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7ffb68074b80 con 0x7ffb4c070d60 2023-10-14T12:00:04.146 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-14T12:00:04.147 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-14T12:00:04.147 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-14T12:00:04.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.141+0000 7ffb6f64f700 1 -- 172.21.15.78:0/1489747491 >> [v2:172.21.15.78:6816/3584112288,v1:172.21.15.78:6817/3584112288] conn(0x7ffb4c070d60 msgr2=0x7ffb4c073180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:04.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.141+0000 7ffb6f64f700 1 --2- 172.21.15.78:0/1489747491 >> [v2:172.21.15.78:6816/3584112288,v1:172.21.15.78:6817/3584112288] conn(0x7ffb4c070d60 0x7ffb4c073180 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.141+0000 7ffb6f64f700 1 -- 172.21.15.78:0/1489747491 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ffb4c06af00 msgr2=0x7ffb4c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:04.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.141+0000 7ffb6f64f700 1 --2- 172.21.15.78:0/1489747491 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ffb4c06af00 0x7ffb4c06d3c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ffb58000c00 tx=0x7ffb58008030 comp rx=0 tx=0).stop 2023-10-14T12:00:04.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.141+0000 7ffb6f64f700 1 -- 172.21.15.78:0/1489747491 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb68152070 msgr2=0x7ffb68151130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:04.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.141+0000 7ffb6f64f700 1 --2- 172.21.15.78:0/1489747491 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb68152070 0x7ffb68151130 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7ffb6400c7f0 tx=0x7ffb6400cbb0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.141+0000 7ffb6dbec700 1 -- 172.21.15.78:0/1489747491 reap_dead start 2023-10-14T12:00:04.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.141+0000 7ffb6f64f700 1 -- 172.21.15.78:0/1489747491 shutdown_connections 2023-10-14T12:00:04.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.141+0000 7ffb6f64f700 1 -- 172.21.15.78:0/1489747491 >> 172.21.15.78:0/1489747491 conn(0x7ffb680712f0 msgr2=0x7ffb6806d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:04.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.141+0000 7ffb6f64f700 1 -- 172.21.15.78:0/1489747491 shutdown_connections 2023-10-14T12:00:04.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.141+0000 7ffb6f64f700 1 -- 172.21.15.78:0/1489747491 wait complete. 2023-10-14T12:00:04.155 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-10-14T12:00:04.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.273+0000 7ff15cf42700 1 Processor -- start 2023-10-14T12:00:04.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.273+0000 7ff15cf42700 1 -- start start 2023-10-14T12:00:04.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.274+0000 7ff15cf42700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff15814ed10 0x7ff158152ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:04.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.274+0000 7ff15cf42700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff158154c20 0x7ff158155030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:04.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.274+0000 7ff15cf42700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1580c26d0 con 0x7ff1580c35e0 2023-10-14T12:00:04.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.274+0000 7ff15cf42700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1580a8520 con 0x7ff15814ed10 2023-10-14T12:00:04.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.274+0000 7ff15cf42700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1580c29b0 con 0x7ff158154c20 2023-10-14T12:00:04.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.274+0000 7ff15659c700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff15814ed10 0x7ff158152ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:04.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.274+0000 7ff15659c700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff15814ed10 0x7ff158152ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47290/0 (socket says 172.21.15.78:47290) 2023-10-14T12:00:04.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.274+0000 7ff15659c700 1 -- 172.21.15.78:0/3405197660 learned_addr learned my addr 172.21.15.78:0/3405197660 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:04.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.274+0000 7ff156d9d700 1 --2- 172.21.15.78:0/3405197660 >> v2:172.21.15.195:3300/0 conn(0x7ff158154c20 0x7ff158155030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:04.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.274+0000 7ff15659c700 1 -- 172.21.15.78:0/3405197660 >> v2:172.21.15.195:3300/0 conn(0x7ff158154c20 msgr2=0x7ff158155030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T12:00:04.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.274+0000 7ff15659c700 1 --2- 172.21.15.78:0/3405197660 >> v2:172.21.15.195:3300/0 conn(0x7ff158154c20 0x7ff158155030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.274+0000 7ff15659c700 1 -- 172.21.15.78:0/3405197660 >> v1:172.21.15.195:6789/0 conn(0x7ff1580c35e0 legacy=0x7ff158153400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T12:00:04.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.274+0000 7ff15659c700 1 -- 172.21.15.78:0/3405197660 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1580c2c90 con 0x7ff15814ed10 2023-10-14T12:00:04.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.274+0000 7ff15659c700 1 --2- 172.21.15.78:0/3405197660 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff15814ed10 0x7ff158152ec0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7ff158072ed0 tx=0x7ff148009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=816df968306ca05e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:04.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.274+0000 7ff15559a700 1 -- 172.21.15.78:0/3405197660 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff148004230 con 0x7ff15814ed10 2023-10-14T12:00:04.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.274+0000 7ff15559a700 1 -- 172.21.15.78:0/3405197660 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff148004390 con 0x7ff15814ed10 2023-10-14T12:00:04.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.274+0000 7ff15559a700 1 -- 172.21.15.78:0/3405197660 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1480036a0 con 0x7ff15814ed10 2023-10-14T12:00:04.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.275+0000 7ff15cf42700 1 -- 172.21.15.78:0/3405197660 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff15814ed10 msgr2=0x7ff158152ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:04.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.275+0000 7ff15cf42700 1 --2- 172.21.15.78:0/3405197660 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff15814ed10 0x7ff158152ec0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7ff158072ed0 tx=0x7ff148009bd0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.275+0000 7ff15cf42700 1 -- 172.21.15.78:0/3405197660 shutdown_connections 2023-10-14T12:00:04.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.275+0000 7ff15cf42700 1 --2- 172.21.15.78:0/3405197660 >> v2:172.21.15.195:3300/0 conn(0x7ff158154c20 0x7ff158155030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.275+0000 7ff15cf42700 1 --2- 172.21.15.78:0/3405197660 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff15814ed10 0x7ff158152ec0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.275+0000 7ff15cf42700 1 -- 172.21.15.78:0/3405197660 >> 172.21.15.78:0/3405197660 conn(0x7ff1580c3a50 msgr2=0x7ff1580c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:04.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.275+0000 7ff15cf42700 1 -- 172.21.15.78:0/3405197660 shutdown_connections 2023-10-14T12:00:04.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.275+0000 7ff15cf42700 1 -- 172.21.15.78:0/3405197660 wait complete. 2023-10-14T12:00:04.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.277+0000 7ff15cf42700 1 Processor -- start 2023-10-14T12:00:04.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.277+0000 7ff15cf42700 1 -- start start 2023-10-14T12:00:04.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.278+0000 7ff15cf42700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff15814ed10 0x7ff1580c5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:04.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.278+0000 7ff15cf42700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff158154c20 0x7ff1580c0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:04.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.278+0000 7ff15cf42700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff158151fc0 0x7ff1580bfe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:04.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.278+0000 7ff15cf42700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1581524c0 con 0x7ff158151fc0 2023-10-14T12:00:04.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.278+0000 7ff15cf42700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1580bdc90 con 0x7ff158154c20 2023-10-14T12:00:04.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.278+0000 7ff15cf42700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1580abda0 con 0x7ff15814ed10 2023-10-14T12:00:04.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.278+0000 7ff156d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff158151fc0 0x7ff1580bfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:04.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.278+0000 7ff156d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff158151fc0 0x7ff1580bfe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47300/0 (socket says 172.21.15.78:47300) 2023-10-14T12:00:04.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.278+0000 7ff156d9d700 1 -- 172.21.15.78:0/1477454531 learned_addr learned my addr 172.21.15.78:0/1477454531 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:04.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.278+0000 7ff15659c700 1 --2- 172.21.15.78:0/1477454531 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff15814ed10 0x7ff1580c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:04.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.278+0000 7ff155d9b700 1 --2- 172.21.15.78:0/1477454531 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff158154c20 0x7ff1580c0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:04.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.278+0000 7ff156d9d700 1 -- 172.21.15.78:0/1477454531 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff15814ed10 msgr2=0x7ff1580c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:04.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.278+0000 7ff156d9d700 1 --2- 172.21.15.78:0/1477454531 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff15814ed10 0x7ff1580c5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.278+0000 7ff156d9d700 1 -- 172.21.15.78:0/1477454531 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff158154c20 msgr2=0x7ff1580c0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:04.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.278+0000 7ff156d9d700 1 --2- 172.21.15.78:0/1477454531 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff158154c20 0x7ff1580c0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.278+0000 7ff156d9d700 1 -- 172.21.15.78:0/1477454531 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff148009550 con 0x7ff158151fc0 2023-10-14T12:00:04.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.278+0000 7ff156d9d700 1 --2- 172.21.15.78:0/1477454531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff158151fc0 0x7ff1580bfe00 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff150007cf0 tx=0x7ff15000e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:04.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.278+0000 7ff146ffd700 1 -- 172.21.15.78:0/1477454531 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff15000eb70 con 0x7ff158151fc0 2023-10-14T12:00:04.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.278+0000 7ff146ffd700 1 -- 172.21.15.78:0/1477454531 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff15000ecf0 con 0x7ff158151fc0 2023-10-14T12:00:04.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.278+0000 7ff15cf42700 1 -- 172.21.15.78:0/1477454531 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1580ac230 con 0x7ff158151fc0 2023-10-14T12:00:04.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.278+0000 7ff146ffd700 1 -- 172.21.15.78:0/1477454531 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff150014660 con 0x7ff158151fc0 2023-10-14T12:00:04.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.278+0000 7ff15cf42700 1 -- 172.21.15.78:0/1477454531 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1580bc5e0 con 0x7ff158151fc0 2023-10-14T12:00:04.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.279+0000 7ff15cf42700 1 -- 172.21.15.78:0/1477454531 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1580abf50 con 0x7ff158151fc0 2023-10-14T12:00:04.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.280+0000 7ff146ffd700 1 -- 172.21.15.78:0/1477454531 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff1500147c0 con 0x7ff158151fc0 2023-10-14T12:00:04.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.280+0000 7ff146ffd700 1 --2- 172.21.15.78:0/1477454531 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff13806af00 0x7ff13806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:04.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.280+0000 7ff146ffd700 1 -- 172.21.15.78:0/1477454531 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff150089840 con 0x7ff158151fc0 2023-10-14T12:00:04.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.280+0000 7ff15659c700 1 --2- 172.21.15.78:0/1477454531 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff13806af00 0x7ff13806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:04.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.281+0000 7ff15659c700 1 --2- 172.21.15.78:0/1477454531 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff13806af00 0x7ff13806d3c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff148009130 tx=0x7ff148008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:04.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.285+0000 7ff146ffd700 1 -- 172.21.15.78:0/1477454531 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff1580abf50 con 0x7ff158151fc0 2023-10-14T12:00:04.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.473+0000 7ff15cf42700 1 -- 172.21.15.78:0/1477454531 --> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7ff158153b10 con 0x7ff13806af00 2023-10-14T12:00:04.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.473+0000 7ff146ffd700 1 -- 172.21.15.78:0/1477454531 <== mgr.4104 v2:172.21.15.78:6832/3171890074 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff148004130 con 0x7ff13806af00 2023-10-14T12:00:04.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.475+0000 7ff15cf42700 1 -- 172.21.15.78:0/1477454531 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff13806af00 msgr2=0x7ff13806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:04.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.475+0000 7ff15cf42700 1 --2- 172.21.15.78:0/1477454531 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff13806af00 0x7ff13806d3c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff148009130 tx=0x7ff148008030 comp rx=0 tx=0).stop 2023-10-14T12:00:04.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.476+0000 7ff15cf42700 1 -- 172.21.15.78:0/1477454531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff158151fc0 msgr2=0x7ff1580bfe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:04.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.476+0000 7ff15cf42700 1 --2- 172.21.15.78:0/1477454531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff158151fc0 0x7ff1580bfe00 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff150007cf0 tx=0x7ff15000e0e0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.476+0000 7ff15cf42700 1 -- 172.21.15.78:0/1477454531 shutdown_connections 2023-10-14T12:00:04.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.476+0000 7ff15cf42700 1 --2- 172.21.15.78:0/1477454531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff158151fc0 0x7ff1580bfe00 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.476+0000 7ff15cf42700 1 --2- 172.21.15.78:0/1477454531 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff15814ed10 0x7ff1580c5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.476+0000 7ff15cf42700 1 --2- 172.21.15.78:0/1477454531 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7ff13806af00 0x7ff13806d3c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.476+0000 7ff15cf42700 1 --2- 172.21.15.78:0/1477454531 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff158154c20 0x7ff1580c0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.476+0000 7ff15cf42700 1 -- 172.21.15.78:0/1477454531 >> 172.21.15.78:0/1477454531 conn(0x7ff1580c3a50 msgr2=0x7ff158153340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:04.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.476+0000 7ff15cf42700 1 -- 172.21.15.78:0/1477454531 shutdown_connections 2023-10-14T12:00:04.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.476+0000 7ff15cf42700 1 -- 172.21.15.78:0/1477454531 wait complete. 2023-10-14T12:00:04.481 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 3 to deep-scrub 2023-10-14T12:00:04.489 INFO:tasks.ceph:Scrubbing osd.4 2023-10-14T12:00:04.490 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-10-14T12:00:04.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.612+0000 7f79d89e8700 1 Processor -- start 2023-10-14T12:00:04.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.612+0000 7f79d89e8700 1 -- start start 2023-10-14T12:00:04.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.613+0000 7f79d89e8700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f79d414feb0 0x7f79d406dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:04.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.613+0000 7f79d89e8700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f79d4071800 0x7f79d4071c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:04.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.613+0000 7f79d89e8700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d4072170 con 0x7f79d406d920 2023-10-14T12:00:04.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.613+0000 7f79d89e8700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79d406e4e0 con 0x7f79d414feb0 2023-10-14T12:00:04.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.613+0000 7f79d89e8700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79d40c66e0 con 0x7f79d4071800 2023-10-14T12:00:04.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.613+0000 7f79d1d9b700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f79d406d920 0x7f79d40c10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52180/0 (socket says 172.21.15.78:52180) 2023-10-14T12:00:04.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.613+0000 7f79d1d9b700 1 -- 172.21.15.78:0/1315078157 learned_addr learned my addr 172.21.15.78:0/1315078157 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T12:00:04.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.613+0000 7f79d2d9d700 1 --2- 172.21.15.78:0/1315078157 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f79d4071800 0x7f79d4071c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:04.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.613+0000 7f79d259c700 1 --2- 172.21.15.78:0/1315078157 >> v2:172.21.15.195:3300/0 conn(0x7f79d414feb0 0x7f79d406dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:04.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.613+0000 7f79d159a700 1 -- 172.21.15.78:0/1315078157 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 760373197 0 0) 0x7f79d4072170 con 0x7f79d406d920 2023-10-14T12:00:04.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.613+0000 7f79d159a700 1 -- 172.21.15.78:0/1315078157 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79bc0035b0 con 0x7f79d406d920 2023-10-14T12:00:04.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.613+0000 7f79d2d9d700 1 -- 172.21.15.78:0/1315078157 >> v2:172.21.15.195:3300/0 conn(0x7f79d414feb0 msgr2=0x7f79d406dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T12:00:04.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.613+0000 7f79d2d9d700 1 --2- 172.21.15.78:0/1315078157 >> v2:172.21.15.195:3300/0 conn(0x7f79d414feb0 0x7f79d406dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.613+0000 7f79d2d9d700 1 -- 172.21.15.78:0/1315078157 >> v1:172.21.15.78:6789/0 conn(0x7f79d406d920 legacy=0x7f79d40c10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:04.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.613+0000 7f79d159a700 1 -- 172.21.15.78:0/1315078157 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3480824720 0 0) 0x7f79bc0035b0 con 0x7f79d406d920 2023-10-14T12:00:04.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.613+0000 7f79d2d9d700 1 -- 172.21.15.78:0/1315078157 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d4150290 con 0x7f79d4071800 2023-10-14T12:00:04.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.614+0000 7f79d2d9d700 1 --2- 172.21.15.78:0/1315078157 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f79d4071800 0x7f79d4071c30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f79cc009d90 tx=0x7f79cc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=782f99a3199101 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:04.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.614+0000 7f79d159a700 1 -- 172.21.15.78:0/1315078157 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79cc00e070 con 0x7f79d4071800 2023-10-14T12:00:04.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.614+0000 7f79d159a700 1 -- 172.21.15.78:0/1315078157 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79cc003a20 con 0x7f79d4071800 2023-10-14T12:00:04.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.614+0000 7f79d159a700 1 -- 172.21.15.78:0/1315078157 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c8001670 con 0x7f79d4071800 2023-10-14T12:00:04.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.614+0000 7f79d89e8700 1 -- 172.21.15.78:0/1315078157 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f79d4071800 msgr2=0x7f79d4071c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:04.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.614+0000 7f79d89e8700 1 --2- 172.21.15.78:0/1315078157 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f79d4071800 0x7f79d4071c30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f79cc009d90 tx=0x7f79cc00b330 comp rx=0 tx=0).stop 2023-10-14T12:00:04.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.614+0000 7f79d89e8700 1 -- 172.21.15.78:0/1315078157 shutdown_connections 2023-10-14T12:00:04.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.614+0000 7f79d89e8700 1 --2- 172.21.15.78:0/1315078157 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f79d4071800 0x7f79d4071c30 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.614+0000 7f79d89e8700 1 --2- 172.21.15.78:0/1315078157 >> v2:172.21.15.195:3300/0 conn(0x7f79d414feb0 0x7f79d406dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.614+0000 7f79d89e8700 1 -- 172.21.15.78:0/1315078157 >> 172.21.15.78:0/1315078157 conn(0x7f79d40742a0 msgr2=0x7f79d40a8490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:04.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.614+0000 7f79d89e8700 1 -- 172.21.15.78:0/1315078157 shutdown_connections 2023-10-14T12:00:04.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.614+0000 7f79d89e8700 1 -- 172.21.15.78:0/1315078157 wait complete. 2023-10-14T12:00:04.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.616+0000 7f79d89e8700 1 Processor -- start 2023-10-14T12:00:04.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.616+0000 7f79d89e8700 1 -- start start 2023-10-14T12:00:04.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.616+0000 7f79d89e8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79d406d920 0x7f79d40bca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:04.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.616+0000 7f79d89e8700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79d4071800 0x7f79d40beca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:04.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.616+0000 7f79d89e8700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f79d414feb0 0x7f79d41eb750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:04.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.616+0000 7f79d89e8700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d40bc020 con 0x7f79d406d920 2023-10-14T12:00:04.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.616+0000 7f79d89e8700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d40bc5e0 con 0x7f79d4071800 2023-10-14T12:00:04.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.616+0000 7f79d89e8700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79d40bddc0 con 0x7f79d414feb0 2023-10-14T12:00:04.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.616+0000 7f79d259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79d406d920 0x7f79d40bca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:04.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.616+0000 7f79d259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79d406d920 0x7f79d40bca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47310/0 (socket says 172.21.15.78:47310) 2023-10-14T12:00:04.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.616+0000 7f79d259c700 1 -- 172.21.15.78:0/4092783940 learned_addr learned my addr 172.21.15.78:0/4092783940 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:04.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.617+0000 7f79d2d9d700 1 --2- 172.21.15.78:0/4092783940 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f79d414feb0 0x7f79d41eb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:04.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.617+0000 7f79d1d9b700 1 --2- 172.21.15.78:0/4092783940 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79d4071800 0x7f79d40beca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:04.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.617+0000 7f79d259c700 1 -- 172.21.15.78:0/4092783940 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f79d414feb0 msgr2=0x7f79d41eb750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:04.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.617+0000 7f79d259c700 1 --2- 172.21.15.78:0/4092783940 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f79d414feb0 0x7f79d41eb750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.617+0000 7f79d259c700 1 -- 172.21.15.78:0/4092783940 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79d4071800 msgr2=0x7f79d40beca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:04.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.617+0000 7f79d259c700 1 --2- 172.21.15.78:0/4092783940 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79d4071800 0x7f79d40beca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.633 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.617+0000 7f79d259c700 1 -- 172.21.15.78:0/4092783940 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79cc009a70 con 0x7f79d406d920 2023-10-14T12:00:04.633 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.617+0000 7f79d2d9d700 1 --2- 172.21.15.78:0/4092783940 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f79d414feb0 0x7f79d41eb750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T12:00:04.633 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.617+0000 7f79d1d9b700 1 --2- 172.21.15.78:0/4092783940 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79d4071800 0x7f79d40beca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T12:00:04.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.617+0000 7f79d259c700 1 --2- 172.21.15.78:0/4092783940 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79d406d920 0x7f79d40bca70 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f79c400e9e0 tx=0x7f79c400eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:04.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.617+0000 7f79c2ffd700 1 -- 172.21.15.78:0/4092783940 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c4009710 con 0x7f79d406d920 2023-10-14T12:00:04.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.617+0000 7f79c2ffd700 1 -- 172.21.15.78:0/4092783940 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79c400c850 con 0x7f79d406d920 2023-10-14T12:00:04.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.617+0000 7f79d89e8700 1 -- 172.21.15.78:0/4092783940 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79d40be0a0 con 0x7f79d406d920 2023-10-14T12:00:04.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.617+0000 7f79c2ffd700 1 -- 172.21.15.78:0/4092783940 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c400cbc0 con 0x7f79d406d920 2023-10-14T12:00:04.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.617+0000 7f79d89e8700 1 -- 172.21.15.78:0/4092783940 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79d4066d80 con 0x7f79d406d920 2023-10-14T12:00:04.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.618+0000 7f79d89e8700 1 -- 172.21.15.78:0/4092783940 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f79d406e9d0 con 0x7f79d406d920 2023-10-14T12:00:04.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.618+0000 7f79c2ffd700 1 -- 172.21.15.78:0/4092783940 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f79c400cd20 con 0x7f79d406d920 2023-10-14T12:00:04.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.619+0000 7f79c2ffd700 1 --2- 172.21.15.78:0/4092783940 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f79b406af00 0x7f79b406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:04.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.619+0000 7f79c2ffd700 1 -- 172.21.15.78:0/4092783940 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f79c40893d0 con 0x7f79d406d920 2023-10-14T12:00:04.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.619+0000 7f79d1d9b700 1 --2- 172.21.15.78:0/4092783940 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f79b406af00 0x7f79b406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:04.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.619+0000 7f79c2ffd700 1 --2- 172.21.15.78:0/4092783940 >> [v2:172.21.15.195:6800/3732756301,v1:172.21.15.195:6801/3732756301] conn(0x7f79b4070d60 0x7f79b4073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:04.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.619+0000 7f79c2ffd700 1 -- 172.21.15.78:0/4092783940 --> [v2:172.21.15.195:6800/3732756301,v1:172.21.15.195:6801/3732756301] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f79b4073850 con 0x7f79b4070d60 2023-10-14T12:00:04.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.619+0000 7f79c2ffd700 1 -- 172.21.15.78:0/4092783940 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f79c408ba90 con 0x7f79d406d920 2023-10-14T12:00:04.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.619+0000 7f79d1d9b700 1 --2- 172.21.15.78:0/4092783940 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f79b406af00 0x7f79b406d3c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f79c8001eb0 tx=0x7f79c8001c60 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:04.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.619+0000 7f79d2d9d700 1 --2- 172.21.15.78:0/4092783940 >> [v2:172.21.15.195:6800/3732756301,v1:172.21.15.195:6801/3732756301] conn(0x7f79b4070d60 0x7f79b4073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:04.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.620+0000 7f79d2d9d700 1 --2- 172.21.15.78:0/4092783940 >> [v2:172.21.15.195:6800/3732756301,v1:172.21.15.195:6801/3732756301] conn(0x7f79b4070d60 0x7f79b4073180 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:04.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.621+0000 7f79c2ffd700 1 -- 172.21.15.78:0/4092783940 <== osd.4 v2:172.21.15.195:6800/3732756301 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f79b4073850 con 0x7f79b4070d60 2023-10-14T12:00:04.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.635+0000 7f79d89e8700 1 -- 172.21.15.78:0/4092783940 --> [v2:172.21.15.195:6800/3732756301,v1:172.21.15.195:6801/3732756301] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f79d406e6f0 con 0x7f79b4070d60 2023-10-14T12:00:04.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.636+0000 7f79c2ffd700 1 -- 172.21.15.78:0/4092783940 <== osd.4 v2:172.21.15.195:6800/3732756301 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f79cc003de0 con 0x7f79b4070d60 2023-10-14T12:00:04.640 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-14T12:00:04.640 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-14T12:00:04.640 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-14T12:00:04.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.636+0000 7f79d89e8700 1 -- 172.21.15.78:0/4092783940 >> [v2:172.21.15.195:6800/3732756301,v1:172.21.15.195:6801/3732756301] conn(0x7f79b4070d60 msgr2=0x7f79b4073180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:04.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.636+0000 7f79d89e8700 1 --2- 172.21.15.78:0/4092783940 >> [v2:172.21.15.195:6800/3732756301,v1:172.21.15.195:6801/3732756301] conn(0x7f79b4070d60 0x7f79b4073180 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.636+0000 7f79d89e8700 1 -- 172.21.15.78:0/4092783940 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f79b406af00 msgr2=0x7f79b406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:04.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.636+0000 7f79d89e8700 1 --2- 172.21.15.78:0/4092783940 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f79b406af00 0x7f79b406d3c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f79c8001eb0 tx=0x7f79c8001c60 comp rx=0 tx=0).stop 2023-10-14T12:00:04.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.636+0000 7f79d89e8700 1 -- 172.21.15.78:0/4092783940 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79d406d920 msgr2=0x7f79d40bca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:04.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.636+0000 7f79d89e8700 1 --2- 172.21.15.78:0/4092783940 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79d406d920 0x7f79d40bca70 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f79c400e9e0 tx=0x7f79c400eda0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.636+0000 7f79d2d9d700 1 -- 172.21.15.78:0/4092783940 reap_dead start 2023-10-14T12:00:04.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.636+0000 7f79d89e8700 1 -- 172.21.15.78:0/4092783940 shutdown_connections 2023-10-14T12:00:04.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.636+0000 7f79d89e8700 1 -- 172.21.15.78:0/4092783940 >> 172.21.15.78:0/4092783940 conn(0x7f79d40742a0 msgr2=0x7f79d4149770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:04.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.636+0000 7f79d89e8700 1 -- 172.21.15.78:0/4092783940 shutdown_connections 2023-10-14T12:00:04.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.636+0000 7f79d89e8700 1 -- 172.21.15.78:0/4092783940 wait complete. 2023-10-14T12:00:04.649 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-10-14T12:00:04.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.769+0000 7fb7a44fb700 1 Processor -- start 2023-10-14T12:00:04.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.769+0000 7fb7a44fb700 1 -- start start 2023-10-14T12:00:04.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.770+0000 7fb7a44fb700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb79c06b500 0x7fb79c0c09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:04.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.770+0000 7fb7a44fb700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb79c06ebe0 con 0x7fb79c0c4540 2023-10-14T12:00:04.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.770+0000 7fb7a44fb700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb79c1558c0 con 0x7fb79c13dbe0 2023-10-14T12:00:04.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.770+0000 7fb7a44fb700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb79c06d330 con 0x7fb79c06b500 2023-10-14T12:00:04.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.770+0000 7fb7a2a98700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb79c06b500 0x7fb79c0c09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:04.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.770+0000 7fb7a1a96700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb79c0c4540 0x7fb79c0c02e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52198/0 (socket says 172.21.15.78:52198) 2023-10-14T12:00:04.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.770+0000 7fb7a1a96700 1 -- 172.21.15.78:0/597186156 learned_addr learned my addr 172.21.15.78:0/597186156 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T12:00:04.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.770+0000 7fb7a2a98700 1 -- 172.21.15.78:0/597186156 >> v1:172.21.15.195:6789/0 conn(0x7fb79c13dbe0 legacy=0x7fb79c06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:04.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.770+0000 7fb7a2a98700 1 -- 172.21.15.78:0/597186156 >> v1:172.21.15.78:6789/0 conn(0x7fb79c0c4540 legacy=0x7fb79c0c02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:04.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.770+0000 7fb7a1295700 1 -- 172.21.15.78:0/597186156 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2813385638 0 0) 0x7fb79c06ebe0 con 0x7fb79c0c4540 2023-10-14T12:00:04.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.770+0000 7fb7a2a98700 1 -- 172.21.15.78:0/597186156 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb79c154b00 con 0x7fb79c06b500 2023-10-14T12:00:04.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.770+0000 7fb7a2a98700 1 --2- 172.21.15.78:0/597186156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb79c06b500 0x7fb79c0c09f0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fb78c009d90 tx=0x7fb78c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78574bc37502b4b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:04.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.771+0000 7fb7a1295700 1 -- 172.21.15.78:0/597186156 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb78c00e070 con 0x7fb79c06b500 2023-10-14T12:00:04.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.771+0000 7fb7a1295700 1 -- 172.21.15.78:0/597186156 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb78c00bbd0 con 0x7fb79c06b500 2023-10-14T12:00:04.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.771+0000 7fb7a1295700 1 -- 172.21.15.78:0/597186156 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb78c012430 con 0x7fb79c06b500 2023-10-14T12:00:04.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.771+0000 7fb7a44fb700 1 -- 172.21.15.78:0/597186156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb79c06b500 msgr2=0x7fb79c0c09f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:04.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.771+0000 7fb7a44fb700 1 --2- 172.21.15.78:0/597186156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb79c06b500 0x7fb79c0c09f0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fb78c009d90 tx=0x7fb78c00b330 comp rx=0 tx=0).stop 2023-10-14T12:00:04.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.771+0000 7fb7a44fb700 1 -- 172.21.15.78:0/597186156 shutdown_connections 2023-10-14T12:00:04.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.771+0000 7fb7a44fb700 1 --2- 172.21.15.78:0/597186156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb79c06b500 0x7fb79c0c09f0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.771+0000 7fb7a44fb700 1 -- 172.21.15.78:0/597186156 >> 172.21.15.78:0/597186156 conn(0x7fb79c0c3a50 msgr2=0x7fb79c0c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:04.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.771+0000 7fb7a44fb700 1 -- 172.21.15.78:0/597186156 shutdown_connections 2023-10-14T12:00:04.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.771+0000 7fb7a44fb700 1 -- 172.21.15.78:0/597186156 wait complete. 2023-10-14T12:00:04.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.772+0000 7fb7a44fb700 1 Processor -- start 2023-10-14T12:00:04.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.772+0000 7fb7a44fb700 1 -- start start 2023-10-14T12:00:04.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.773+0000 7fb7a44fb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb79c06b500 0x7fb79c1566e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:04.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.773+0000 7fb7a44fb700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb79c0c4540 0x7fb79c137a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:04.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.773+0000 7fb7a44fb700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb79c13dbe0 0x7fb79c137f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:04.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.773+0000 7fb7a44fb700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb79c138490 con 0x7fb79c06b500 2023-10-14T12:00:04.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.773+0000 7fb7a44fb700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb79c1386f0 con 0x7fb79c13dbe0 2023-10-14T12:00:04.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.773+0000 7fb7a44fb700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb79c155e10 con 0x7fb79c0c4540 2023-10-14T12:00:04.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.773+0000 7fb7a2297700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb79c06b500 0x7fb79c1566e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:04.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.773+0000 7fb7a2297700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb79c06b500 0x7fb79c1566e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47326/0 (socket says 172.21.15.78:47326) 2023-10-14T12:00:04.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.773+0000 7fb7a2297700 1 -- 172.21.15.78:0/2703123902 learned_addr learned my addr 172.21.15.78:0/2703123902 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:04.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.773+0000 7fb7a1a96700 1 --2- 172.21.15.78:0/2703123902 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb79c0c4540 0x7fb79c137a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:04.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.773+0000 7fb7a2a98700 1 --2- 172.21.15.78:0/2703123902 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb79c13dbe0 0x7fb79c137f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:04.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.773+0000 7fb7a2297700 1 -- 172.21.15.78:0/2703123902 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb79c0c4540 msgr2=0x7fb79c137a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:04.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.773+0000 7fb7a2297700 1 --2- 172.21.15.78:0/2703123902 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb79c0c4540 0x7fb79c137a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.773+0000 7fb7a2297700 1 -- 172.21.15.78:0/2703123902 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb79c13dbe0 msgr2=0x7fb79c137f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:04.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.773+0000 7fb7a2297700 1 --2- 172.21.15.78:0/2703123902 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb79c13dbe0 0x7fb79c137f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.773+0000 7fb7a2297700 1 -- 172.21.15.78:0/2703123902 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb78c009a70 con 0x7fb79c06b500 2023-10-14T12:00:04.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.774+0000 7fb7a2297700 1 --2- 172.21.15.78:0/2703123902 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb79c06b500 0x7fb79c1566e0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fb79400c4e0 tx=0x7fb79400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:04.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.774+0000 7fb79affd700 1 -- 172.21.15.78:0/2703123902 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7940052d0 con 0x7fb79c06b500 2023-10-14T12:00:04.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.774+0000 7fb79affd700 1 -- 172.21.15.78:0/2703123902 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb794005430 con 0x7fb79c06b500 2023-10-14T12:00:04.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.774+0000 7fb7a44fb700 1 -- 172.21.15.78:0/2703123902 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb79c157200 con 0x7fb79c06b500 2023-10-14T12:00:04.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.774+0000 7fb79affd700 1 -- 172.21.15.78:0/2703123902 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7940038c0 con 0x7fb79c06b500 2023-10-14T12:00:04.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.774+0000 7fb7a44fb700 1 -- 172.21.15.78:0/2703123902 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb79c138f50 con 0x7fb79c06b500 2023-10-14T12:00:04.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.775+0000 7fb7a44fb700 1 -- 172.21.15.78:0/2703123902 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb79c074b80 con 0x7fb79c06b500 2023-10-14T12:00:04.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.775+0000 7fb79affd700 1 -- 172.21.15.78:0/2703123902 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb794003a20 con 0x7fb79c06b500 2023-10-14T12:00:04.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.775+0000 7fb79affd700 1 --2- 172.21.15.78:0/2703123902 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb78806af00 0x7fb78806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:04.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.775+0000 7fb79affd700 1 -- 172.21.15.78:0/2703123902 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb794088730 con 0x7fb79c06b500 2023-10-14T12:00:04.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.775+0000 7fb7a1a96700 1 --2- 172.21.15.78:0/2703123902 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb78806af00 0x7fb78806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:04.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.776+0000 7fb7a1a96700 1 --2- 172.21.15.78:0/2703123902 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb78806af00 0x7fb78806d3c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb790007fd0 tx=0x7fb790007c20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:04.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.780+0000 7fb79affd700 1 -- 172.21.15.78:0/2703123902 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb794053b70 con 0x7fb79c06b500 2023-10-14T12:00:04.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.968+0000 7fb7a44fb700 1 -- 172.21.15.78:0/2703123902 --> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fb79c1550c0 con 0x7fb78806af00 2023-10-14T12:00:04.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.968+0000 7fb79affd700 1 -- 172.21.15.78:0/2703123902 <== mgr.4104 v2:172.21.15.78:6832/3171890074 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb790005a20 con 0x7fb78806af00 2023-10-14T12:00:04.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.970+0000 7fb7a44fb700 1 -- 172.21.15.78:0/2703123902 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb78806af00 msgr2=0x7fb78806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:04.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.970+0000 7fb7a44fb700 1 --2- 172.21.15.78:0/2703123902 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb78806af00 0x7fb78806d3c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb790007fd0 tx=0x7fb790007c20 comp rx=0 tx=0).stop 2023-10-14T12:00:04.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.970+0000 7fb7a44fb700 1 -- 172.21.15.78:0/2703123902 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb79c06b500 msgr2=0x7fb79c1566e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:04.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.970+0000 7fb7a44fb700 1 --2- 172.21.15.78:0/2703123902 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb79c06b500 0x7fb79c1566e0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fb79400c4e0 tx=0x7fb79400c8a0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.970+0000 7fb7a44fb700 1 -- 172.21.15.78:0/2703123902 shutdown_connections 2023-10-14T12:00:04.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.970+0000 7fb7a44fb700 1 --2- 172.21.15.78:0/2703123902 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fb78806af00 0x7fb78806d3c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.970+0000 7fb7a44fb700 1 --2- 172.21.15.78:0/2703123902 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb79c13dbe0 0x7fb79c137f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.970+0000 7fb7a44fb700 1 --2- 172.21.15.78:0/2703123902 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb79c06b500 0x7fb79c1566e0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.970+0000 7fb7a44fb700 1 --2- 172.21.15.78:0/2703123902 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb79c0c4540 0x7fb79c137a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:04.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.970+0000 7fb7a44fb700 1 -- 172.21.15.78:0/2703123902 >> 172.21.15.78:0/2703123902 conn(0x7fb79c0c3a50 msgr2=0x7fb79c0c32c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:04.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.970+0000 7fb7a44fb700 1 -- 172.21.15.78:0/2703123902 shutdown_connections 2023-10-14T12:00:04.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:04.970+0000 7fb7a44fb700 1 -- 172.21.15.78:0/2703123902 wait complete. 2023-10-14T12:00:04.976 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 4 to deep-scrub 2023-10-14T12:00:04.984 INFO:tasks.ceph:Scrubbing osd.5 2023-10-14T12:00:04.984 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-10-14T12:00:05.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.107+0000 7f95cd0a0700 1 Processor -- start 2023-10-14T12:00:05.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.107+0000 7f95cd0a0700 1 -- start start 2023-10-14T12:00:05.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.107+0000 7f95cd0a0700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f95c806b490 0x7f95c8160220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:05.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.107+0000 7f95cd0a0700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95c80acae0 con 0x7f95c806e410 2023-10-14T12:00:05.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.107+0000 7f95cd0a0700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95c80ad010 con 0x7f95c8152070 2023-10-14T12:00:05.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.107+0000 7f95cd0a0700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95c806d650 con 0x7f95c806b490 2023-10-14T12:00:05.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.108+0000 7f95c759e700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f95c806b490 0x7f95c8160220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:05.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.108+0000 7f95c659c700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f95c806e410 0x7f95c8071e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52212/0 (socket says 172.21.15.78:52212) 2023-10-14T12:00:05.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.108+0000 7f95c659c700 1 -- 172.21.15.78:0/2385799321 learned_addr learned my addr 172.21.15.78:0/2385799321 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T12:00:05.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.108+0000 7f95c759e700 1 -- 172.21.15.78:0/2385799321 >> v1:172.21.15.195:6789/0 conn(0x7f95c8152070 legacy=0x7f95c8157e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:05.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.108+0000 7f95c759e700 1 -- 172.21.15.78:0/2385799321 >> v1:172.21.15.78:6789/0 conn(0x7f95c806e410 legacy=0x7f95c8071e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:05.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.108+0000 7f95c759e700 1 -- 172.21.15.78:0/2385799321 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c80acd30 con 0x7f95c806b490 2023-10-14T12:00:05.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.108+0000 7f95c5d9b700 1 -- 172.21.15.78:0/2385799321 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 960557993 0 0) 0x7f95c80acae0 con 0x7f95c806e410 2023-10-14T12:00:05.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.108+0000 7f95c759e700 1 --2- 172.21.15.78:0/2385799321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95c806b490 0x7f95c8160220 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f95c0009d90 tx=0x7f95c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f64e6c5ad421b217 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:05.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.108+0000 7f95c5d9b700 1 -- 172.21.15.78:0/2385799321 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c000e070 con 0x7f95c806b490 2023-10-14T12:00:05.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.108+0000 7f95c5d9b700 1 -- 172.21.15.78:0/2385799321 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95c000bbd0 con 0x7f95c806b490 2023-10-14T12:00:05.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.108+0000 7f95c5d9b700 1 -- 172.21.15.78:0/2385799321 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c0012430 con 0x7f95c806b490 2023-10-14T12:00:05.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.109+0000 7f95cd0a0700 1 -- 172.21.15.78:0/2385799321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95c806b490 msgr2=0x7f95c8160220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:05.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.109+0000 7f95cd0a0700 1 --2- 172.21.15.78:0/2385799321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95c806b490 0x7f95c8160220 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f95c0009d90 tx=0x7f95c000b330 comp rx=0 tx=0).stop 2023-10-14T12:00:05.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.109+0000 7f95cd0a0700 1 -- 172.21.15.78:0/2385799321 shutdown_connections 2023-10-14T12:00:05.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.109+0000 7f95cd0a0700 1 --2- 172.21.15.78:0/2385799321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95c806b490 0x7f95c8160220 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:05.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.109+0000 7f95cd0a0700 1 -- 172.21.15.78:0/2385799321 >> 172.21.15.78:0/2385799321 conn(0x7f95c80712f0 msgr2=0x7f95c8071700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:05.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.109+0000 7f95cd0a0700 1 -- 172.21.15.78:0/2385799321 shutdown_connections 2023-10-14T12:00:05.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.109+0000 7f95cd0a0700 1 -- 172.21.15.78:0/2385799321 wait complete. 2023-10-14T12:00:05.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.110+0000 7f95cd0a0700 1 Processor -- start 2023-10-14T12:00:05.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.110+0000 7f95cd0a0700 1 -- start start 2023-10-14T12:00:05.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.111+0000 7f95cd0a0700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f95c806b490 0x7f95c80c2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:05.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.111+0000 7f95cd0a0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95c806e410 0x7f95c8144650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:05.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.111+0000 7f95cd0a0700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f95c8152070 0x7f95c8144b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:05.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.111+0000 7f95cd0a0700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c81529f0 con 0x7f95c806e410 2023-10-14T12:00:05.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.111+0000 7f95cd0a0700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c8151170 con 0x7f95c8152070 2023-10-14T12:00:05.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.111+0000 7f95cd0a0700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95c8151380 con 0x7f95c806b490 2023-10-14T12:00:05.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.111+0000 7f95c659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95c806e410 0x7f95c8144650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:05.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.111+0000 7f95c659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95c806e410 0x7f95c8144650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47342/0 (socket says 172.21.15.78:47342) 2023-10-14T12:00:05.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.111+0000 7f95c659c700 1 -- 172.21.15.78:0/2894076207 learned_addr learned my addr 172.21.15.78:0/2894076207 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:05.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.111+0000 7f95c759e700 1 --2- 172.21.15.78:0/2894076207 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f95c8152070 0x7f95c8144b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:05.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.111+0000 7f95c6d9d700 1 --2- 172.21.15.78:0/2894076207 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f95c806b490 0x7f95c80c2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:05.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.111+0000 7f95c659c700 1 -- 172.21.15.78:0/2894076207 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f95c806b490 msgr2=0x7f95c80c2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:05.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.111+0000 7f95c659c700 1 --2- 172.21.15.78:0/2894076207 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f95c806b490 0x7f95c80c2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:05.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.111+0000 7f95c659c700 1 -- 172.21.15.78:0/2894076207 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f95c8152070 msgr2=0x7f95c8144b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:05.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.111+0000 7f95c659c700 1 --2- 172.21.15.78:0/2894076207 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f95c8152070 0x7f95c8144b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:05.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.111+0000 7f95c659c700 1 -- 172.21.15.78:0/2894076207 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c0009a70 con 0x7f95c806e410 2023-10-14T12:00:05.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.111+0000 7f95c659c700 1 --2- 172.21.15.78:0/2894076207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95c806e410 0x7f95c8144650 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f95bc00c450 tx=0x7f95bc00c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:05.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.111+0000 7f95af7fe700 1 -- 172.21.15.78:0/2894076207 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95bc004a80 con 0x7f95c806e410 2023-10-14T12:00:05.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.112+0000 7f95af7fe700 1 -- 172.21.15.78:0/2894076207 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95bc004c00 con 0x7f95c806e410 2023-10-14T12:00:05.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.112+0000 7f95cd0a0700 1 -- 172.21.15.78:0/2894076207 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95c8152710 con 0x7f95c806e410 2023-10-14T12:00:05.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.112+0000 7f95af7fe700 1 -- 172.21.15.78:0/2894076207 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95bc0070d0 con 0x7f95c806e410 2023-10-14T12:00:05.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.112+0000 7f95cd0a0700 1 -- 172.21.15.78:0/2894076207 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95c8155f70 con 0x7f95c806e410 2023-10-14T12:00:05.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.112+0000 7f95cd0a0700 1 -- 172.21.15.78:0/2894076207 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f95c8158590 con 0x7f95c806e410 2023-10-14T12:00:05.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.113+0000 7f95af7fe700 1 -- 172.21.15.78:0/2894076207 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f95bc007230 con 0x7f95c806e410 2023-10-14T12:00:05.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.113+0000 7f95af7fe700 1 --2- 172.21.15.78:0/2894076207 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f95b406af00 0x7f95b406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:05.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.113+0000 7f95af7fe700 1 -- 172.21.15.78:0/2894076207 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f95bc088d90 con 0x7f95c806e410 2023-10-14T12:00:05.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.113+0000 7f95c6d9d700 1 --2- 172.21.15.78:0/2894076207 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f95b406af00 0x7f95b406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:05.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.113+0000 7f95af7fe700 1 --2- 172.21.15.78:0/2894076207 >> [v2:172.21.15.195:6824/2857706411,v1:172.21.15.195:6825/2857706411] conn(0x7f95b4070d60 0x7f95b4073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:05.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.113+0000 7f95af7fe700 1 -- 172.21.15.78:0/2894076207 --> [v2:172.21.15.195:6824/2857706411,v1:172.21.15.195:6825/2857706411] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95b4073850 con 0x7f95b4070d60 2023-10-14T12:00:05.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.113+0000 7f95af7fe700 1 -- 172.21.15.78:0/2894076207 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f95bc08b450 con 0x7f95c806e410 2023-10-14T12:00:05.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.114+0000 7f95c759e700 1 --2- 172.21.15.78:0/2894076207 >> [v2:172.21.15.195:6824/2857706411,v1:172.21.15.195:6825/2857706411] conn(0x7f95b4070d60 0x7f95b4073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:05.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.114+0000 7f95c6d9d700 1 --2- 172.21.15.78:0/2894076207 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f95b406af00 0x7f95b406d3c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f95c8151830 tx=0x7f95b8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:05.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.114+0000 7f95c759e700 1 --2- 172.21.15.78:0/2894076207 >> [v2:172.21.15.195:6824/2857706411,v1:172.21.15.195:6825/2857706411] conn(0x7f95b4070d60 0x7f95b4073180 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:05.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.115+0000 7f95af7fe700 1 -- 172.21.15.78:0/2894076207 <== osd.5 v2:172.21.15.195:6824/2857706411 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f95b4073850 con 0x7f95b4070d60 2023-10-14T12:00:05.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.129+0000 7f95cd0a0700 1 -- 172.21.15.78:0/2894076207 --> [v2:172.21.15.195:6824/2857706411,v1:172.21.15.195:6825/2857706411] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f95c80c1180 con 0x7f95b4070d60 2023-10-14T12:00:05.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.130+0000 7f95af7fe700 1 -- 172.21.15.78:0/2894076207 <== osd.5 v2:172.21.15.195:6824/2857706411 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f95c0012880 con 0x7f95b4070d60 2023-10-14T12:00:05.132 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-14T12:00:05.132 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-14T12:00:05.133 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-14T12:00:05.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.130+0000 7f95cd0a0700 1 -- 172.21.15.78:0/2894076207 >> [v2:172.21.15.195:6824/2857706411,v1:172.21.15.195:6825/2857706411] conn(0x7f95b4070d60 msgr2=0x7f95b4073180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:05.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.130+0000 7f95cd0a0700 1 --2- 172.21.15.78:0/2894076207 >> [v2:172.21.15.195:6824/2857706411,v1:172.21.15.195:6825/2857706411] conn(0x7f95b4070d60 0x7f95b4073180 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:05.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.130+0000 7f95cd0a0700 1 -- 172.21.15.78:0/2894076207 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f95b406af00 msgr2=0x7f95b406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:05.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.130+0000 7f95cd0a0700 1 --2- 172.21.15.78:0/2894076207 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f95b406af00 0x7f95b406d3c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f95c8151830 tx=0x7f95b8009040 comp rx=0 tx=0).stop 2023-10-14T12:00:05.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.130+0000 7f95cd0a0700 1 -- 172.21.15.78:0/2894076207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95c806e410 msgr2=0x7f95c8144650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:05.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.130+0000 7f95cd0a0700 1 --2- 172.21.15.78:0/2894076207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95c806e410 0x7f95c8144650 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f95bc00c450 tx=0x7f95bc00c760 comp rx=0 tx=0).stop 2023-10-14T12:00:05.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.130+0000 7f95c759e700 1 -- 172.21.15.78:0/2894076207 reap_dead start 2023-10-14T12:00:05.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.130+0000 7f95cd0a0700 1 -- 172.21.15.78:0/2894076207 shutdown_connections 2023-10-14T12:00:05.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.130+0000 7f95cd0a0700 1 -- 172.21.15.78:0/2894076207 >> 172.21.15.78:0/2894076207 conn(0x7f95c80712f0 msgr2=0x7f95c806d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:05.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.130+0000 7f95cd0a0700 1 -- 172.21.15.78:0/2894076207 shutdown_connections 2023-10-14T12:00:05.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.130+0000 7f95cd0a0700 1 -- 172.21.15.78:0/2894076207 wait complete. 2023-10-14T12:00:05.144 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-10-14T12:00:05.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.262+0000 7f0fc22ef700 1 Processor -- start 2023-10-14T12:00:05.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.262+0000 7f0fc22ef700 1 -- start start 2023-10-14T12:00:05.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.263+0000 7f0fc22ef700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0fbc14ed10 0x7f0fbc152ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:05.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.263+0000 7f0fc22ef700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0fbc154c20 0x7f0fbc155090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:05.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.263+0000 7f0fc22ef700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fbc0c26d0 con 0x7f0fbc0c35e0 2023-10-14T12:00:05.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.263+0000 7f0fc22ef700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fbc0a8520 con 0x7f0fbc154c20 2023-10-14T12:00:05.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.263+0000 7f0fc22ef700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fbc0c29b0 con 0x7f0fbc14ed10 2023-10-14T12:00:05.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.263+0000 7f0fbb7fe700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f0fbc0c35e0 0x7f0fbc153400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52228/0 (socket says 172.21.15.78:52228) 2023-10-14T12:00:05.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.263+0000 7f0fbb7fe700 1 -- 172.21.15.78:0/1441883193 learned_addr learned my addr 172.21.15.78:0/1441883193 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T12:00:05.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.263+0000 7f0fc088c700 1 --2- 172.21.15.78:0/1441883193 >> v2:172.21.15.78:3300/0 conn(0x7f0fbc154c20 0x7f0fbc155090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:05.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.263+0000 7f0fbbfff700 1 --2- 172.21.15.78:0/1441883193 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0fbc14ed10 0x7f0fbc152ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:05.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.263+0000 7f0fbaffd700 1 -- 172.21.15.78:0/1441883193 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 378154332 0 0) 0x7f0fbc0c26d0 con 0x7f0fbc0c35e0 2023-10-14T12:00:05.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.264+0000 7f0fbaffd700 1 -- 172.21.15.78:0/1441883193 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fa40035b0 con 0x7f0fbc0c35e0 2023-10-14T12:00:05.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.264+0000 7f0fc088c700 1 -- 172.21.15.78:0/1441883193 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0fbc14ed10 msgr2=0x7f0fbc152ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T12:00:05.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.264+0000 7f0fc088c700 1 --2- 172.21.15.78:0/1441883193 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0fbc14ed10 0x7f0fbc152ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:05.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.264+0000 7f0fc088c700 1 -- 172.21.15.78:0/1441883193 >> v1:172.21.15.78:6789/0 conn(0x7f0fbc0c35e0 legacy=0x7f0fbc153400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:05.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.264+0000 7f0fc088c700 1 -- 172.21.15.78:0/1441883193 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fbc0c2c90 con 0x7f0fbc154c20 2023-10-14T12:00:05.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.264+0000 7f0fc088c700 1 --2- 172.21.15.78:0/1441883193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fbc154c20 0x7f0fbc155090 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4009d90 tx=0x7f0fb400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd92d04c6d0b31da server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:05.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.264+0000 7f0fbaffd700 1 -- 172.21.15.78:0/1441883193 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb400e070 con 0x7f0fbc154c20 2023-10-14T12:00:05.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.264+0000 7f0fbaffd700 1 -- 172.21.15.78:0/1441883193 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fb400bb20 con 0x7f0fbc154c20 2023-10-14T12:00:05.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.264+0000 7f0fbaffd700 1 -- 172.21.15.78:0/1441883193 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb400be90 con 0x7f0fbc154c20 2023-10-14T12:00:05.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.264+0000 7f0fc22ef700 1 -- 172.21.15.78:0/1441883193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fbc154c20 msgr2=0x7f0fbc155090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:05.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.264+0000 7f0fc22ef700 1 --2- 172.21.15.78:0/1441883193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fbc154c20 0x7f0fbc155090 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4009d90 tx=0x7f0fb400b330 comp rx=0 tx=0).stop 2023-10-14T12:00:05.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.265+0000 7f0fc22ef700 1 -- 172.21.15.78:0/1441883193 shutdown_connections 2023-10-14T12:00:05.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.265+0000 7f0fc22ef700 1 --2- 172.21.15.78:0/1441883193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fbc154c20 0x7f0fbc155090 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:05.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.265+0000 7f0fc22ef700 1 --2- 172.21.15.78:0/1441883193 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0fbc14ed10 0x7f0fbc152ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:05.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.265+0000 7f0fc22ef700 1 -- 172.21.15.78:0/1441883193 >> 172.21.15.78:0/1441883193 conn(0x7f0fbc0c3a50 msgr2=0x7f0fbc0c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:05.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.265+0000 7f0fc22ef700 1 -- 172.21.15.78:0/1441883193 shutdown_connections 2023-10-14T12:00:05.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.265+0000 7f0fc22ef700 1 -- 172.21.15.78:0/1441883193 wait complete. 2023-10-14T12:00:05.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.266+0000 7f0fc22ef700 1 Processor -- start 2023-10-14T12:00:05.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.266+0000 7f0fc22ef700 1 -- start start 2023-10-14T12:00:05.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.266+0000 7f0fc22ef700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0fbc14ed10 0x7f0fbc0a9f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:05.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.266+0000 7f0fc22ef700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0fbc154c20 0x7f0fbc1e7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:05.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.266+0000 7f0fc22ef700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fbc06f1f0 0x7f0fbc155030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:05.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.267+0000 7f0fc22ef700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fbc0c5c30 con 0x7f0fbc06f1f0 2023-10-14T12:00:05.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.267+0000 7f0fc22ef700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fbc14f7b0 con 0x7f0fbc154c20 2023-10-14T12:00:05.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.267+0000 7f0fc22ef700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fbc0a9470 con 0x7f0fbc14ed10 2023-10-14T12:00:05.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.267+0000 7f0fc088c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fbc06f1f0 0x7f0fbc155030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:05.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.267+0000 7f0fc088c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fbc06f1f0 0x7f0fbc155030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47360/0 (socket says 172.21.15.78:47360) 2023-10-14T12:00:05.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.267+0000 7f0fc088c700 1 -- 172.21.15.78:0/173379994 learned_addr learned my addr 172.21.15.78:0/173379994 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:05.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.267+0000 7f0fbbfff700 1 --2- 172.21.15.78:0/173379994 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0fbc14ed10 0x7f0fbc0a9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:05.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.267+0000 7f0fbb7fe700 1 --2- 172.21.15.78:0/173379994 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0fbc154c20 0x7f0fbc1e7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:05.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.267+0000 7f0fc088c700 1 -- 172.21.15.78:0/173379994 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0fbc14ed10 msgr2=0x7f0fbc0a9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:05.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.267+0000 7f0fc088c700 1 --2- 172.21.15.78:0/173379994 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0fbc14ed10 0x7f0fbc0a9f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:05.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.267+0000 7f0fc088c700 1 -- 172.21.15.78:0/173379994 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0fbc154c20 msgr2=0x7f0fbc1e7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:05.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.267+0000 7f0fc088c700 1 --2- 172.21.15.78:0/173379994 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0fbc154c20 0x7f0fbc1e7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:05.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.267+0000 7f0fc088c700 1 -- 172.21.15.78:0/173379994 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb4009a70 con 0x7f0fbc06f1f0 2023-10-14T12:00:05.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.267+0000 7f0fbb7fe700 1 --2- 172.21.15.78:0/173379994 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0fbc154c20 0x7f0fbc1e7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T12:00:05.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.267+0000 7f0fc088c700 1 --2- 172.21.15.78:0/173379994 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fbc06f1f0 0x7f0fbc155030 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4014040 tx=0x7f0fb4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:05.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.267+0000 7f0fb8ff9700 1 -- 172.21.15.78:0/173379994 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb4007000 con 0x7f0fbc06f1f0 2023-10-14T12:00:05.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.267+0000 7f0fb8ff9700 1 -- 172.21.15.78:0/173379994 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fb4012860 con 0x7f0fbc06f1f0 2023-10-14T12:00:05.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.267+0000 7f0fb8ff9700 1 -- 172.21.15.78:0/173379994 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb4012c10 con 0x7f0fbc06f1f0 2023-10-14T12:00:05.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.267+0000 7f0fc22ef700 1 -- 172.21.15.78:0/173379994 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fbc0a9750 con 0x7f0fbc06f1f0 2023-10-14T12:00:05.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.267+0000 7f0fc22ef700 1 -- 172.21.15.78:0/173379994 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fbc1e74e0 con 0x7f0fbc06f1f0 2023-10-14T12:00:05.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.268+0000 7f0fc22ef700 1 -- 172.21.15.78:0/173379994 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fbc074b80 con 0x7f0fbc06f1f0 2023-10-14T12:00:05.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.269+0000 7f0fb8ff9700 1 -- 172.21.15.78:0/173379994 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0fb401c400 con 0x7f0fbc06f1f0 2023-10-14T12:00:05.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.269+0000 7f0fb8ff9700 1 --2- 172.21.15.78:0/173379994 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f0fa806af00 0x7f0fa806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:05.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.269+0000 7f0fb8ff9700 1 -- 172.21.15.78:0/173379994 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0fb408b0c0 con 0x7f0fbc06f1f0 2023-10-14T12:00:05.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.269+0000 7f0fbbfff700 1 --2- 172.21.15.78:0/173379994 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f0fa806af00 0x7f0fa806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:05.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.269+0000 7f0fbbfff700 1 --2- 172.21.15.78:0/173379994 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f0fa806af00 0x7f0fa806d3c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0009930 tx=0x7f0fb0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:05.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.273+0000 7f0fb8ff9700 1 -- 172.21.15.78:0/173379994 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0fb40565b0 con 0x7f0fbc06f1f0 2023-10-14T12:00:05.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.462+0000 7f0fc22ef700 1 -- 172.21.15.78:0/173379994 --> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f0fbc153b10 con 0x7f0fa806af00 2023-10-14T12:00:05.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.462+0000 7f0fb8ff9700 1 -- 172.21.15.78:0/173379994 <== mgr.4104 v2:172.21.15.78:6832/3171890074 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0fb0009e00 con 0x7f0fa806af00 2023-10-14T12:00:05.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.464+0000 7f0fc22ef700 1 -- 172.21.15.78:0/173379994 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f0fa806af00 msgr2=0x7f0fa806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:05.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.464+0000 7f0fc22ef700 1 --2- 172.21.15.78:0/173379994 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f0fa806af00 0x7f0fa806d3c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0009930 tx=0x7f0fb0008040 comp rx=0 tx=0).stop 2023-10-14T12:00:05.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.464+0000 7f0fc22ef700 1 -- 172.21.15.78:0/173379994 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fbc06f1f0 msgr2=0x7f0fbc155030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:05.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.464+0000 7f0fc22ef700 1 --2- 172.21.15.78:0/173379994 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fbc06f1f0 0x7f0fbc155030 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4014040 tx=0x7f0fb4004ad0 comp rx=0 tx=0).stop 2023-10-14T12:00:05.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.464+0000 7f0fc22ef700 1 -- 172.21.15.78:0/173379994 shutdown_connections 2023-10-14T12:00:05.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.464+0000 7f0fc22ef700 1 --2- 172.21.15.78:0/173379994 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fbc06f1f0 0x7f0fbc155030 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:05.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.464+0000 7f0fc22ef700 1 --2- 172.21.15.78:0/173379994 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0fbc14ed10 0x7f0fbc0a9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:05.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.464+0000 7f0fc22ef700 1 --2- 172.21.15.78:0/173379994 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f0fa806af00 0x7f0fa806d3c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:05.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.464+0000 7f0fc22ef700 1 --2- 172.21.15.78:0/173379994 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0fbc154c20 0x7f0fbc1e7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:05.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.464+0000 7f0fc22ef700 1 -- 172.21.15.78:0/173379994 >> 172.21.15.78:0/173379994 conn(0x7f0fbc0c3a50 msgr2=0x7f0fbc153d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:05.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.464+0000 7f0fc22ef700 1 -- 172.21.15.78:0/173379994 shutdown_connections 2023-10-14T12:00:05.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.464+0000 7f0fc22ef700 1 -- 172.21.15.78:0/173379994 wait complete. 2023-10-14T12:00:05.470 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 5 to deep-scrub 2023-10-14T12:00:05.478 INFO:tasks.ceph:Scrubbing osd.6 2023-10-14T12:00:05.478 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-10-14T12:00:05.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.601+0000 7f19f303a700 1 Processor -- start 2023-10-14T12:00:05.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.601+0000 7f19f303a700 1 -- start start 2023-10-14T12:00:05.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.601+0000 7f19f303a700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f19ec0c1340 0x7f19ec0c1720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:05.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.601+0000 7f19f303a700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f19ec0c2120 0x7f19ec14ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:05.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.601+0000 7f19f303a700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ec1556b0 con 0x7f19ec14efd0 2023-10-14T12:00:05.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.601+0000 7f19f303a700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19ec0c1c60 con 0x7f19ec0c1340 2023-10-14T12:00:05.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.601+0000 7f19f303a700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19ec073a90 con 0x7f19ec0c2120 2023-10-14T12:00:05.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.602+0000 7f19f0dd6700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f19ec0c1340 0x7f19ec0c1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:05.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.602+0000 7f19f0dd6700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f19ec0c1340 0x7f19ec0c1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47364/0 (socket says 172.21.15.78:47364) 2023-10-14T12:00:05.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.602+0000 7f19f0dd6700 1 -- 172.21.15.78:0/28907909 learned_addr learned my addr 172.21.15.78:0/28907909 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:05.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.602+0000 7f19ebfff700 1 --2- 172.21.15.78:0/28907909 >> v2:172.21.15.195:3300/0 conn(0x7f19ec0c2120 0x7f19ec14ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:05.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.602+0000 7f19f0dd6700 1 -- 172.21.15.78:0/28907909 >> v2:172.21.15.195:3300/0 conn(0x7f19ec0c2120 msgr2=0x7f19ec14ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T12:00:05.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.602+0000 7f19f0dd6700 1 --2- 172.21.15.78:0/28907909 >> v2:172.21.15.195:3300/0 conn(0x7f19ec0c2120 0x7f19ec14ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:05.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.602+0000 7f19f0dd6700 1 -- 172.21.15.78:0/28907909 >> v1:172.21.15.195:6789/0 conn(0x7f19ec14efd0 legacy=0x7f19ec0a9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:05.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.602+0000 7f19f0dd6700 1 -- 172.21.15.78:0/28907909 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ec155920 con 0x7f19ec0c1340 2023-10-14T12:00:05.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.602+0000 7f19f0dd6700 1 --2- 172.21.15.78:0/28907909 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19ec0c1340 0x7f19ec0c1720 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f19ec06ecd0 tx=0x7f19e0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22253afaebf1e23b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:05.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.602+0000 7f19eb7fe700 1 -- 172.21.15.78:0/28907909 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e0011830 con 0x7f19ec0c1340 2023-10-14T12:00:05.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.602+0000 7f19eb7fe700 1 -- 172.21.15.78:0/28907909 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e0011990 con 0x7f19ec0c1340 2023-10-14T12:00:05.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.602+0000 7f19eb7fe700 1 -- 172.21.15.78:0/28907909 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e0011d20 con 0x7f19ec0c1340 2023-10-14T12:00:05.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.603+0000 7f19f303a700 1 -- 172.21.15.78:0/28907909 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19ec0c1340 msgr2=0x7f19ec0c1720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:05.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.603+0000 7f19f303a700 1 --2- 172.21.15.78:0/28907909 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19ec0c1340 0x7f19ec0c1720 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f19ec06ecd0 tx=0x7f19e0009bd0 comp rx=0 tx=0).stop 2023-10-14T12:00:05.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.603+0000 7f19f303a700 1 -- 172.21.15.78:0/28907909 shutdown_connections 2023-10-14T12:00:05.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.603+0000 7f19f303a700 1 --2- 172.21.15.78:0/28907909 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19ec0c1340 0x7f19ec0c1720 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:05.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.603+0000 7f19f303a700 1 --2- 172.21.15.78:0/28907909 >> v2:172.21.15.195:3300/0 conn(0x7f19ec0c2120 0x7f19ec14ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:05.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.603+0000 7f19f303a700 1 -- 172.21.15.78:0/28907909 >> 172.21.15.78:0/28907909 conn(0x7f19ec0742a0 msgr2=0x7f19ec0a8490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:05.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.603+0000 7f19f303a700 1 -- 172.21.15.78:0/28907909 shutdown_connections 2023-10-14T12:00:05.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.603+0000 7f19f303a700 1 -- 172.21.15.78:0/28907909 wait complete. 2023-10-14T12:00:05.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.604+0000 7f19f303a700 1 Processor -- start 2023-10-14T12:00:05.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.604+0000 7f19f303a700 1 -- start start 2023-10-14T12:00:05.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.605+0000 7f19f303a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19ec0c1340 0x7f19ec1e6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:05.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.605+0000 7f19f303a700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f19ec0c2120 0x7f19ec14dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:05.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.605+0000 7f19f303a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f19ec14efd0 0x7f19ec0bf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:05.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.605+0000 7f19f303a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec1ea9c0 con 0x7f19ec0c1340 2023-10-14T12:00:05.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.605+0000 7f19f303a700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec0a8840 con 0x7f19ec14efd0 2023-10-14T12:00:05.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.605+0000 7f19f303a700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec1e9d40 con 0x7f19ec0c2120 2023-10-14T12:00:05.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.605+0000 7f19f0dd6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19ec0c1340 0x7f19ec1e6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:05.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.605+0000 7f19f0dd6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19ec0c1340 0x7f19ec1e6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47370/0 (socket says 172.21.15.78:47370) 2023-10-14T12:00:05.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.605+0000 7f19f0dd6700 1 -- 172.21.15.78:0/2921217225 learned_addr learned my addr 172.21.15.78:0/2921217225 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:05.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.605+0000 7f19f15d7700 1 --2- 172.21.15.78:0/2921217225 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f19ec14efd0 0x7f19ec0bf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:05.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.605+0000 7f19ebfff700 1 --2- 172.21.15.78:0/2921217225 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f19ec0c2120 0x7f19ec14dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:05.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.605+0000 7f19f0dd6700 1 -- 172.21.15.78:0/2921217225 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f19ec0c2120 msgr2=0x7f19ec14dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:05.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.605+0000 7f19f0dd6700 1 --2- 172.21.15.78:0/2921217225 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f19ec0c2120 0x7f19ec14dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:05.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.605+0000 7f19f0dd6700 1 -- 172.21.15.78:0/2921217225 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f19ec14efd0 msgr2=0x7f19ec0bf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:05.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.605+0000 7f19f0dd6700 1 --2- 172.21.15.78:0/2921217225 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f19ec14efd0 0x7f19ec0bf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:05.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.605+0000 7f19f0dd6700 1 -- 172.21.15.78:0/2921217225 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e0009550 con 0x7f19ec0c1340 2023-10-14T12:00:05.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.605+0000 7f19f0dd6700 1 --2- 172.21.15.78:0/2921217225 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19ec0c1340 0x7f19ec1e6460 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f19e0000c00 tx=0x7f19e0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:05.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.605+0000 7f19e97fa700 1 -- 172.21.15.78:0/2921217225 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e0004570 con 0x7f19ec0c1340 2023-10-14T12:00:05.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.605+0000 7f19e97fa700 1 -- 172.21.15.78:0/2921217225 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e001f070 con 0x7f19ec0c1340 2023-10-14T12:00:05.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.605+0000 7f19f303a700 1 -- 172.21.15.78:0/2921217225 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19ec1ea020 con 0x7f19ec0c1340 2023-10-14T12:00:05.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.605+0000 7f19e97fa700 1 -- 172.21.15.78:0/2921217225 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e00038e0 con 0x7f19ec0c1340 2023-10-14T12:00:05.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.606+0000 7f19f303a700 1 -- 172.21.15.78:0/2921217225 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19ec1ea5e0 con 0x7f19ec0c1340 2023-10-14T12:00:05.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.606+0000 7f19f303a700 1 -- 172.21.15.78:0/2921217225 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f19ec0c1e50 con 0x7f19ec0c1340 2023-10-14T12:00:05.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.607+0000 7f19e97fa700 1 -- 172.21.15.78:0/2921217225 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f19e0021030 con 0x7f19ec0c1340 2023-10-14T12:00:05.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.607+0000 7f19e97fa700 1 --2- 172.21.15.78:0/2921217225 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f19d006af00 0x7f19d006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:05.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.607+0000 7f19e97fa700 1 -- 172.21.15.78:0/2921217225 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f19e008a5d0 con 0x7f19ec0c1340 2023-10-14T12:00:05.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.607+0000 7f19ebfff700 1 --2- 172.21.15.78:0/2921217225 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f19d006af00 0x7f19d006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:05.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.607+0000 7f19e97fa700 1 --2- 172.21.15.78:0/2921217225 >> [v2:172.21.15.195:6808/72554374,v1:172.21.15.195:6810/72554374] conn(0x7f19d0070d60 0x7f19d0073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:05.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.607+0000 7f19e97fa700 1 -- 172.21.15.78:0/2921217225 --> [v2:172.21.15.195:6808/72554374,v1:172.21.15.195:6810/72554374] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19d0073850 con 0x7f19d0070d60 2023-10-14T12:00:05.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.607+0000 7f19e97fa700 1 -- 172.21.15.78:0/2921217225 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f19e008cc90 con 0x7f19ec0c1340 2023-10-14T12:00:05.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.608+0000 7f19f15d7700 1 --2- 172.21.15.78:0/2921217225 >> [v2:172.21.15.195:6808/72554374,v1:172.21.15.195:6810/72554374] conn(0x7f19d0070d60 0x7f19d0073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:05.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.608+0000 7f19ebfff700 1 --2- 172.21.15.78:0/2921217225 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f19d006af00 0x7f19d006d3c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f19ec1480b0 tx=0x7f19d8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:05.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.608+0000 7f19f15d7700 1 --2- 172.21.15.78:0/2921217225 >> [v2:172.21.15.195:6808/72554374,v1:172.21.15.195:6810/72554374] conn(0x7f19d0070d60 0x7f19d0073180 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:05.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.609+0000 7f19e97fa700 1 -- 172.21.15.78:0/2921217225 <== osd.6 v2:172.21.15.195:6808/72554374 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f19d0073850 con 0x7f19d0070d60 2023-10-14T12:00:05.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.623+0000 7f19f303a700 1 -- 172.21.15.78:0/2921217225 --> [v2:172.21.15.195:6808/72554374,v1:172.21.15.195:6810/72554374] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f19ec06dac0 con 0x7f19d0070d60 2023-10-14T12:00:05.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.624+0000 7f19e97fa700 1 -- 172.21.15.78:0/2921217225 <== osd.6 v2:172.21.15.195:6808/72554374 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f19e4003a00 con 0x7f19d0070d60 2023-10-14T12:00:05.626 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-14T12:00:05.626 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-14T12:00:05.627 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-14T12:00:05.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.624+0000 7f19f303a700 1 -- 172.21.15.78:0/2921217225 >> [v2:172.21.15.195:6808/72554374,v1:172.21.15.195:6810/72554374] conn(0x7f19d0070d60 msgr2=0x7f19d0073180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:05.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.624+0000 7f19f303a700 1 --2- 172.21.15.78:0/2921217225 >> [v2:172.21.15.195:6808/72554374,v1:172.21.15.195:6810/72554374] conn(0x7f19d0070d60 0x7f19d0073180 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:05.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.624+0000 7f19f303a700 1 -- 172.21.15.78:0/2921217225 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f19d006af00 msgr2=0x7f19d006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:05.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.624+0000 7f19f303a700 1 --2- 172.21.15.78:0/2921217225 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f19d006af00 0x7f19d006d3c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f19ec1480b0 tx=0x7f19d8008040 comp rx=0 tx=0).stop 2023-10-14T12:00:05.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.624+0000 7f19f303a700 1 -- 172.21.15.78:0/2921217225 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19ec0c1340 msgr2=0x7f19ec1e6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:05.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.624+0000 7f19f303a700 1 --2- 172.21.15.78:0/2921217225 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19ec0c1340 0x7f19ec1e6460 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f19e0000c00 tx=0x7f19e0004750 comp rx=0 tx=0).stop 2023-10-14T12:00:05.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.624+0000 7f19f15d7700 1 -- 172.21.15.78:0/2921217225 reap_dead start 2023-10-14T12:00:05.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.624+0000 7f19f303a700 1 -- 172.21.15.78:0/2921217225 shutdown_connections 2023-10-14T12:00:05.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.624+0000 7f19f303a700 1 -- 172.21.15.78:0/2921217225 >> 172.21.15.78:0/2921217225 conn(0x7f19ec0742a0 msgr2=0x7f19ec066a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:05.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.624+0000 7f19f303a700 1 -- 172.21.15.78:0/2921217225 shutdown_connections 2023-10-14T12:00:05.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.624+0000 7f19f303a700 1 -- 172.21.15.78:0/2921217225 wait complete. 2023-10-14T12:00:05.638 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-10-14T12:00:05.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.757+0000 7f2cba0b7700 1 Processor -- start 2023-10-14T12:00:05.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.757+0000 7f2cba0b7700 1 -- start start 2023-10-14T12:00:05.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.758+0000 7f2cba0b7700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2cb414e990 0x7f2cb406e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:05.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.758+0000 7f2cba0b7700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2cb40c0c80 0x7f2cb406ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:05.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.758+0000 7f2cba0b7700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb40c1180 con 0x7f2cb40c4540 2023-10-14T12:00:05.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.758+0000 7f2cba0b7700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cb4155410 con 0x7f2cb40c0c80 2023-10-14T12:00:05.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.758+0000 7f2cba0b7700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb41558a0 con 0x7f2cb414e990 2023-10-14T12:00:05.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.758+0000 7f2cb2ffd700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2cb40c0c80 0x7f2cb406ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:05.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.758+0000 7f2cb2ffd700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2cb40c0c80 0x7f2cb406ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47382/0 (socket says 172.21.15.78:47382) 2023-10-14T12:00:05.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.758+0000 7f2cb2ffd700 1 -- 172.21.15.78:0/608339456 learned_addr learned my addr 172.21.15.78:0/608339456 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:05.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.758+0000 7f2cb37fe700 1 --2- 172.21.15.78:0/608339456 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2cb414e990 0x7f2cb406e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:05.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.758+0000 7f2cb2ffd700 1 -- 172.21.15.78:0/608339456 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2cb414e990 msgr2=0x7f2cb406e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T12:00:05.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.758+0000 7f2cb2ffd700 1 --2- 172.21.15.78:0/608339456 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2cb414e990 0x7f2cb406e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:05.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.758+0000 7f2cb2ffd700 1 -- 172.21.15.78:0/608339456 >> v1:172.21.15.195:6789/0 conn(0x7f2cb40c4540 legacy=0x7f2cb4157f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T12:00:05.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.758+0000 7f2cb2ffd700 1 -- 172.21.15.78:0/608339456 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb4154ae0 con 0x7f2cb40c0c80 2023-10-14T12:00:05.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.758+0000 7f2cb2ffd700 1 --2- 172.21.15.78:0/608339456 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cb40c0c80 0x7f2cb406ec10 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f2ca40098f0 tx=0x7f2ca4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34ca58be4de7dc04 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:05.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.758+0000 7f2cb27fc700 1 -- 172.21.15.78:0/608339456 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca4004230 con 0x7f2cb40c0c80 2023-10-14T12:00:05.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.759+0000 7f2cb27fc700 1 -- 172.21.15.78:0/608339456 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ca4004390 con 0x7f2cb40c0c80 2023-10-14T12:00:05.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.759+0000 7f2cb27fc700 1 -- 172.21.15.78:0/608339456 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca40036a0 con 0x7f2cb40c0c80 2023-10-14T12:00:05.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.759+0000 7f2cba0b7700 1 -- 172.21.15.78:0/608339456 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cb40c0c80 msgr2=0x7f2cb406ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:05.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.759+0000 7f2cba0b7700 1 --2- 172.21.15.78:0/608339456 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cb40c0c80 0x7f2cb406ec10 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f2ca40098f0 tx=0x7f2ca4009c00 comp rx=0 tx=0).stop 2023-10-14T12:00:05.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.759+0000 7f2cba0b7700 1 -- 172.21.15.78:0/608339456 shutdown_connections 2023-10-14T12:00:05.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.759+0000 7f2cba0b7700 1 --2- 172.21.15.78:0/608339456 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2cb414e990 0x7f2cb406e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:05.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.759+0000 7f2cba0b7700 1 --2- 172.21.15.78:0/608339456 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cb40c0c80 0x7f2cb406ec10 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:05.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.759+0000 7f2cba0b7700 1 -- 172.21.15.78:0/608339456 >> 172.21.15.78:0/608339456 conn(0x7f2cb40c3a20 msgr2=0x7f2cb40c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:05.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.759+0000 7f2cba0b7700 1 -- 172.21.15.78:0/608339456 shutdown_connections 2023-10-14T12:00:05.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.759+0000 7f2cba0b7700 1 -- 172.21.15.78:0/608339456 wait complete. 2023-10-14T12:00:05.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.760+0000 7f2cba0b7700 1 Processor -- start 2023-10-14T12:00:05.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.760+0000 7f2cba0b7700 1 -- start start 2023-10-14T12:00:05.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.761+0000 7f2cba0b7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cb40c0c80 0x7f2cb41edc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:05.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.761+0000 7f2cba0b7700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2cb40c4540 0x7f2cb414c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:05.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.761+0000 7f2cba0b7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2cb414e990 0x7f2cb4146e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:05.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.761+0000 7f2cba0b7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb41ed7e0 con 0x7f2cb40c0c80 2023-10-14T12:00:05.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.761+0000 7f2cba0b7700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb4149a00 con 0x7f2cb414e990 2023-10-14T12:00:05.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.761+0000 7f2cba0b7700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb41ed1d0 con 0x7f2cb40c4540 2023-10-14T12:00:05.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.761+0000 7f2cb37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cb40c0c80 0x7f2cb41edc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:05.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.761+0000 7f2cb37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cb40c0c80 0x7f2cb41edc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47386/0 (socket says 172.21.15.78:47386) 2023-10-14T12:00:05.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.761+0000 7f2cb37fe700 1 -- 172.21.15.78:0/260273541 learned_addr learned my addr 172.21.15.78:0/260273541 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:05.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.761+0000 7f2cb2ffd700 1 --2- 172.21.15.78:0/260273541 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2cb40c4540 0x7f2cb414c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:05.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.761+0000 7f2cb37fe700 1 -- 172.21.15.78:0/260273541 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2cb40c4540 msgr2=0x7f2cb414c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:05.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.761+0000 7f2cb37fe700 1 --2- 172.21.15.78:0/260273541 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2cb40c4540 0x7f2cb414c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:05.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.761+0000 7f2cb3fff700 1 --2- 172.21.15.78:0/260273541 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2cb414e990 0x7f2cb4146e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:05.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.761+0000 7f2cb37fe700 1 -- 172.21.15.78:0/260273541 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2cb414e990 msgr2=0x7f2cb4146e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:05.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.761+0000 7f2cb37fe700 1 --2- 172.21.15.78:0/260273541 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2cb414e990 0x7f2cb4146e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:05.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.761+0000 7f2cb37fe700 1 -- 172.21.15.78:0/260273541 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca4009550 con 0x7f2cb40c0c80 2023-10-14T12:00:05.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.762+0000 7f2cb37fe700 1 --2- 172.21.15.78:0/260273541 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cb40c0c80 0x7f2cb41edc70 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f2ca800c7f0 tx=0x7f2ca800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:05.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.762+0000 7f2c9bfff700 1 -- 172.21.15.78:0/260273541 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca8011800 con 0x7f2cb40c0c80 2023-10-14T12:00:05.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.762+0000 7f2c9bfff700 1 -- 172.21.15.78:0/260273541 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ca8011960 con 0x7f2cb40c0c80 2023-10-14T12:00:05.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.762+0000 7f2cba0b7700 1 -- 172.21.15.78:0/260273541 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cb4149720 con 0x7f2cb40c0c80 2023-10-14T12:00:05.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.762+0000 7f2c9bfff700 1 -- 172.21.15.78:0/260273541 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca8011cf0 con 0x7f2cb40c0c80 2023-10-14T12:00:05.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.762+0000 7f2cba0b7700 1 -- 172.21.15.78:0/260273541 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cb414cb60 con 0x7f2cb40c0c80 2023-10-14T12:00:05.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.763+0000 7f2cba0b7700 1 -- 172.21.15.78:0/260273541 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cb41edac0 con 0x7f2cb40c0c80 2023-10-14T12:00:05.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.763+0000 7f2c9bfff700 1 -- 172.21.15.78:0/260273541 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2ca8018360 con 0x7f2cb40c0c80 2023-10-14T12:00:05.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.763+0000 7f2c9bfff700 1 --2- 172.21.15.78:0/260273541 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f2ca006af00 0x7f2ca006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:05.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.764+0000 7f2c9bfff700 1 -- 172.21.15.78:0/260273541 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2ca80891d0 con 0x7f2cb40c0c80 2023-10-14T12:00:05.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.764+0000 7f2cb2ffd700 1 --2- 172.21.15.78:0/260273541 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f2ca006af00 0x7f2ca006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:05.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.764+0000 7f2cb2ffd700 1 --2- 172.21.15.78:0/260273541 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f2ca006af00 0x7f2ca006d3c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2cb40c2300 tx=0x7f2ca4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:05.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.768+0000 7f2c9bfff700 1 -- 172.21.15.78:0/260273541 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2ca8054610 con 0x7f2cb40c0c80 2023-10-14T12:00:05.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.960+0000 7f2cba0b7700 1 -- 172.21.15.78:0/260273541 --> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f2cb41550a0 con 0x7f2ca006af00 2023-10-14T12:00:05.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.960+0000 7f2c9bfff700 1 -- 172.21.15.78:0/260273541 <== mgr.4104 v2:172.21.15.78:6832/3171890074 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2ca4020070 con 0x7f2ca006af00 2023-10-14T12:00:05.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.962+0000 7f2cba0b7700 1 -- 172.21.15.78:0/260273541 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f2ca006af00 msgr2=0x7f2ca006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:05.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.962+0000 7f2cba0b7700 1 --2- 172.21.15.78:0/260273541 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f2ca006af00 0x7f2ca006d3c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2cb40c2300 tx=0x7f2ca4008030 comp rx=0 tx=0).stop 2023-10-14T12:00:05.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.962+0000 7f2cba0b7700 1 -- 172.21.15.78:0/260273541 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cb40c0c80 msgr2=0x7f2cb41edc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:05.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.962+0000 7f2cba0b7700 1 --2- 172.21.15.78:0/260273541 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cb40c0c80 0x7f2cb41edc70 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f2ca800c7f0 tx=0x7f2ca800cbb0 comp rx=0 tx=0).stop 2023-10-14T12:00:05.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.962+0000 7f2cba0b7700 1 -- 172.21.15.78:0/260273541 shutdown_connections 2023-10-14T12:00:05.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.962+0000 7f2cba0b7700 1 --2- 172.21.15.78:0/260273541 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f2ca006af00 0x7f2ca006d3c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:05.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.962+0000 7f2cba0b7700 1 --2- 172.21.15.78:0/260273541 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2cb414e990 0x7f2cb4146e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:05.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.962+0000 7f2cba0b7700 1 --2- 172.21.15.78:0/260273541 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cb40c0c80 0x7f2cb41edc70 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:05.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.962+0000 7f2cba0b7700 1 --2- 172.21.15.78:0/260273541 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2cb40c4540 0x7f2cb414c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:05.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.962+0000 7f2cba0b7700 1 -- 172.21.15.78:0/260273541 >> 172.21.15.78:0/260273541 conn(0x7f2cb40c3a20 msgr2=0x7f2cb406e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:05.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.962+0000 7f2cba0b7700 1 -- 172.21.15.78:0/260273541 shutdown_connections 2023-10-14T12:00:05.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:05.962+0000 7f2cba0b7700 1 -- 172.21.15.78:0/260273541 wait complete. 2023-10-14T12:00:05.968 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 6 to deep-scrub 2023-10-14T12:00:05.976 INFO:tasks.ceph:Scrubbing osd.7 2023-10-14T12:00:05.976 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-10-14T12:00:06.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.098+0000 7f5c7d49b700 1 Processor -- start 2023-10-14T12:00:06.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.098+0000 7f5c7d49b700 1 -- start start 2023-10-14T12:00:06.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.099+0000 7f5c7d49b700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5c780a72c0 0x7f5c78158a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:06.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.099+0000 7f5c7d49b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f5c7806d070 0x7f5c780c7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:06.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.099+0000 7f5c7d49b700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c78158fd0 con 0x7f5c780701c0 2023-10-14T12:00:06.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.099+0000 7f5c7d49b700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c7806fde0 con 0x7f5c7806d070 2023-10-14T12:00:06.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.099+0000 7f5c7d49b700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c78158670 con 0x7f5c780a72c0 2023-10-14T12:00:06.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.099+0000 7f5c76ffd700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5c780a72c0 0x7f5c78158a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:06.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.099+0000 7f5c777fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f5c7806d070 0x7f5c780c7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:06.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.099+0000 7f5c777fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f5c7806d070 0x7f5c780c7ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.78:50136/0 (socket says 172.21.15.78:50136) 2023-10-14T12:00:06.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.099+0000 7f5c777fe700 1 -- 172.21.15.78:0/3473424404 learned_addr learned my addr 172.21.15.78:0/3473424404 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:06.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.100+0000 7f5c777fe700 1 -- 172.21.15.78:0/3473424404 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5c780a72c0 msgr2=0x7f5c78158a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T12:00:06.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.100+0000 7f5c777fe700 1 --2- 172.21.15.78:0/3473424404 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5c780a72c0 0x7f5c78158a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:06.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.100+0000 7f5c777fe700 1 -- 172.21.15.78:0/3473424404 >> v1:172.21.15.195:6789/0 conn(0x7f5c780701c0 legacy=0x7f5c78154b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:06.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.100+0000 7f5c777fe700 1 -- 172.21.15.78:0/3473424404 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c7806f820 con 0x7f5c7806d070 2023-10-14T12:00:06.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.100+0000 7f5c76ffd700 1 --2- 172.21.15.78:0/3473424404 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5c780a72c0 0x7f5c78158a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T12:00:06.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.100+0000 7f5c777fe700 1 --2- 172.21.15.78:0/3473424404 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5c7806d070 0x7f5c780c7ff0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5c70009e50 tx=0x7f5c7000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d071626e463b1e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:06.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.100+0000 7f5c75ffb700 1 -- 172.21.15.78:0/3473424404 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7000e070 con 0x7f5c7806d070 2023-10-14T12:00:06.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.100+0000 7f5c75ffb700 1 -- 172.21.15.78:0/3473424404 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c7000b910 con 0x7f5c7806d070 2023-10-14T12:00:06.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.100+0000 7f5c75ffb700 1 -- 172.21.15.78:0/3473424404 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7000bcc0 con 0x7f5c7806d070 2023-10-14T12:00:06.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.101+0000 7f5c7d49b700 1 -- 172.21.15.78:0/3473424404 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5c7806d070 msgr2=0x7f5c780c7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:06.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.101+0000 7f5c7d49b700 1 --2- 172.21.15.78:0/3473424404 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5c7806d070 0x7f5c780c7ff0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5c70009e50 tx=0x7f5c7000b330 comp rx=0 tx=0).stop 2023-10-14T12:00:06.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.101+0000 7f5c7d49b700 1 -- 172.21.15.78:0/3473424404 shutdown_connections 2023-10-14T12:00:06.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.101+0000 7f5c7d49b700 1 --2- 172.21.15.78:0/3473424404 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5c7806d070 0x7f5c780c7ff0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:06.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.101+0000 7f5c7d49b700 1 --2- 172.21.15.78:0/3473424404 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5c780a72c0 0x7f5c78158a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:06.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.101+0000 7f5c7d49b700 1 -- 172.21.15.78:0/3473424404 >> 172.21.15.78:0/3473424404 conn(0x7f5c78072790 msgr2=0x7f5c780733e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:06.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.101+0000 7f5c7d49b700 1 -- 172.21.15.78:0/3473424404 shutdown_connections 2023-10-14T12:00:06.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.101+0000 7f5c7d49b700 1 -- 172.21.15.78:0/3473424404 wait complete. 2023-10-14T12:00:06.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.102+0000 7f5c7d49b700 1 Processor -- start 2023-10-14T12:00:06.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.102+0000 7f5c7d49b700 1 -- start start 2023-10-14T12:00:06.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.102+0000 7f5c7d49b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5c7806d070 0x7f5c781eaac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:06.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.102+0000 7f5c7d49b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c780701c0 0x7f5c780bf130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:06.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.103+0000 7f5c7d49b700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5c780a72c0 0x7f5c78153dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:06.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.103+0000 7f5c7d49b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c780bf7c0 con 0x7f5c780701c0 2023-10-14T12:00:06.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.103+0000 7f5c7d49b700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c781e9fe0 con 0x7f5c7806d070 2023-10-14T12:00:06.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.103+0000 7f5c7d49b700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c781ea2c0 con 0x7f5c780a72c0 2023-10-14T12:00:06.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.103+0000 7f5c767fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c780701c0 0x7f5c780bf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:06.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.103+0000 7f5c767fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c780701c0 0x7f5c780bf130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47402/0 (socket says 172.21.15.78:47402) 2023-10-14T12:00:06.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.103+0000 7f5c767fc700 1 -- 172.21.15.78:0/3993608093 learned_addr learned my addr 172.21.15.78:0/3993608093 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:06.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.103+0000 7f5c76ffd700 1 --2- 172.21.15.78:0/3993608093 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5c7806d070 0x7f5c781eaac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:06.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.103+0000 7f5c777fe700 1 --2- 172.21.15.78:0/3993608093 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5c780a72c0 0x7f5c78153dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:06.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.103+0000 7f5c767fc700 1 -- 172.21.15.78:0/3993608093 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5c780a72c0 msgr2=0x7f5c78153dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:06.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.103+0000 7f5c767fc700 1 --2- 172.21.15.78:0/3993608093 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5c780a72c0 0x7f5c78153dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:06.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.103+0000 7f5c767fc700 1 -- 172.21.15.78:0/3993608093 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5c7806d070 msgr2=0x7f5c781eaac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:06.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.103+0000 7f5c767fc700 1 --2- 172.21.15.78:0/3993608093 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5c7806d070 0x7f5c781eaac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:06.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.103+0000 7f5c767fc700 1 -- 172.21.15.78:0/3993608093 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c6c0094a0 con 0x7f5c780701c0 2023-10-14T12:00:06.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.103+0000 7f5c767fc700 1 --2- 172.21.15.78:0/3993608093 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c780701c0 0x7f5c780bf130 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f5c6800c430 tx=0x7f5c6800c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:06.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.103+0000 7f5c5f7fe700 1 -- 172.21.15.78:0/3993608093 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c680050d0 con 0x7f5c780701c0 2023-10-14T12:00:06.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.103+0000 7f5c5f7fe700 1 -- 172.21.15.78:0/3993608093 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c68005250 con 0x7f5c780701c0 2023-10-14T12:00:06.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.103+0000 7f5c7d49b700 1 -- 172.21.15.78:0/3993608093 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c70009b30 con 0x7f5c780701c0 2023-10-14T12:00:06.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.103+0000 7f5c5f7fe700 1 -- 172.21.15.78:0/3993608093 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c68006860 con 0x7f5c780701c0 2023-10-14T12:00:06.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.103+0000 7f5c7d49b700 1 -- 172.21.15.78:0/3993608093 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c780c6cf0 con 0x7f5c780701c0 2023-10-14T12:00:06.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.104+0000 7f5c7d49b700 1 -- 172.21.15.78:0/3993608093 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5c78158290 con 0x7f5c780701c0 2023-10-14T12:00:06.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.105+0000 7f5c5f7fe700 1 -- 172.21.15.78:0/3993608093 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f5c68003680 con 0x7f5c780701c0 2023-10-14T12:00:06.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.105+0000 7f5c5f7fe700 1 --2- 172.21.15.78:0/3993608093 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f5c6406af00 0x7f5c6406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:06.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.105+0000 7f5c5f7fe700 1 -- 172.21.15.78:0/3993608093 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5c68088320 con 0x7f5c780701c0 2023-10-14T12:00:06.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.105+0000 7f5c76ffd700 1 --2- 172.21.15.78:0/3993608093 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f5c6406af00 0x7f5c6406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:06.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.105+0000 7f5c5f7fe700 1 --2- 172.21.15.78:0/3993608093 >> [v2:172.21.15.195:6809/482472979,v1:172.21.15.195:6811/482472979] conn(0x7f5c64070d60 0x7f5c64073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:06.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.105+0000 7f5c5f7fe700 1 -- 172.21.15.78:0/3993608093 --> [v2:172.21.15.195:6809/482472979,v1:172.21.15.195:6811/482472979] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5c64073850 con 0x7f5c64070d60 2023-10-14T12:00:06.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.105+0000 7f5c5f7fe700 1 -- 172.21.15.78:0/3993608093 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5c6808a9e0 con 0x7f5c780701c0 2023-10-14T12:00:06.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.105+0000 7f5c76ffd700 1 --2- 172.21.15.78:0/3993608093 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f5c6406af00 0x7f5c6406d3c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c009d70 tx=0x7f5c6c009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:06.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.106+0000 7f5c777fe700 1 --2- 172.21.15.78:0/3993608093 >> [v2:172.21.15.195:6809/482472979,v1:172.21.15.195:6811/482472979] conn(0x7f5c64070d60 0x7f5c64073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:06.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.106+0000 7f5c777fe700 1 --2- 172.21.15.78:0/3993608093 >> [v2:172.21.15.195:6809/482472979,v1:172.21.15.195:6811/482472979] conn(0x7f5c64070d60 0x7f5c64073180 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:06.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.107+0000 7f5c5f7fe700 1 -- 172.21.15.78:0/3993608093 <== osd.7 v2:172.21.15.195:6809/482472979 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f5c64073850 con 0x7f5c64070d60 2023-10-14T12:00:06.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.121+0000 7f5c7d49b700 1 -- 172.21.15.78:0/3993608093 --> [v2:172.21.15.195:6809/482472979,v1:172.21.15.195:6811/482472979] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f5c7806fb00 con 0x7f5c64070d60 2023-10-14T12:00:06.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.122+0000 7f5c5f7fe700 1 -- 172.21.15.78:0/3993608093 <== osd.7 v2:172.21.15.195:6809/482472979 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f5c700092e0 con 0x7f5c64070d60 2023-10-14T12:00:06.125 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-14T12:00:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-14T12:00:06.126 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-14T12:00:06.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.122+0000 7f5c7d49b700 1 -- 172.21.15.78:0/3993608093 >> [v2:172.21.15.195:6809/482472979,v1:172.21.15.195:6811/482472979] conn(0x7f5c64070d60 msgr2=0x7f5c64073180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:06.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.122+0000 7f5c7d49b700 1 --2- 172.21.15.78:0/3993608093 >> [v2:172.21.15.195:6809/482472979,v1:172.21.15.195:6811/482472979] conn(0x7f5c64070d60 0x7f5c64073180 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:06.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.123+0000 7f5c7d49b700 1 -- 172.21.15.78:0/3993608093 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f5c6406af00 msgr2=0x7f5c6406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:06.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.123+0000 7f5c7d49b700 1 --2- 172.21.15.78:0/3993608093 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f5c6406af00 0x7f5c6406d3c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c009d70 tx=0x7f5c6c009210 comp rx=0 tx=0).stop 2023-10-14T12:00:06.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.123+0000 7f5c7d49b700 1 -- 172.21.15.78:0/3993608093 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c780701c0 msgr2=0x7f5c780bf130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:06.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.123+0000 7f5c7d49b700 1 --2- 172.21.15.78:0/3993608093 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c780701c0 0x7f5c780bf130 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f5c6800c430 tx=0x7f5c6800c740 comp rx=0 tx=0).stop 2023-10-14T12:00:06.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.123+0000 7f5c777fe700 1 -- 172.21.15.78:0/3993608093 reap_dead start 2023-10-14T12:00:06.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.123+0000 7f5c7d49b700 1 -- 172.21.15.78:0/3993608093 shutdown_connections 2023-10-14T12:00:06.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.123+0000 7f5c7d49b700 1 -- 172.21.15.78:0/3993608093 >> 172.21.15.78:0/3993608093 conn(0x7f5c78072790 msgr2=0x7f5c7806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:06.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.123+0000 7f5c7d49b700 1 -- 172.21.15.78:0/3993608093 shutdown_connections 2023-10-14T12:00:06.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.123+0000 7f5c7d49b700 1 -- 172.21.15.78:0/3993608093 wait complete. 2023-10-14T12:00:06.136 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-10-14T12:00:06.257 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.255+0000 7f25ae523700 1 Processor -- start 2023-10-14T12:00:06.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.255+0000 7f25ae523700 1 -- start start 2023-10-14T12:00:06.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.256+0000 7f25ae523700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f25a80a70c0 0x7f25a8154960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:06.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.256+0000 7f25ae523700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f25a8154ea0 0x7f25a815d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:06.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.256+0000 7f25ae523700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a81537b0 con 0x7f25a81533d0 2023-10-14T12:00:06.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.256+0000 7f25ae523700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25a80c6fb0 con 0x7f25a80a70c0 2023-10-14T12:00:06.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.256+0000 7f25ae523700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25a80c7f30 con 0x7f25a8154ea0 2023-10-14T12:00:06.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.256+0000 7f25a77fe700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f25a80a70c0 0x7f25a8154960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:06.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.256+0000 7f25a77fe700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f25a80a70c0 0x7f25a8154960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47412/0 (socket says 172.21.15.78:47412) 2023-10-14T12:00:06.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.256+0000 7f25a77fe700 1 -- 172.21.15.78:0/3880671363 learned_addr learned my addr 172.21.15.78:0/3880671363 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:06.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.256+0000 7f25acac0700 1 --2- 172.21.15.78:0/3880671363 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f25a8154ea0 0x7f25a815d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:06.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.256+0000 7f25a77fe700 1 -- 172.21.15.78:0/3880671363 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f25a8154ea0 msgr2=0x7f25a815d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T12:00:06.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.256+0000 7f25a77fe700 1 --2- 172.21.15.78:0/3880671363 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f25a8154ea0 0x7f25a815d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:06.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.256+0000 7f25a77fe700 1 -- 172.21.15.78:0/3880671363 >> v1:172.21.15.195:6789/0 conn(0x7f25a81533d0 legacy=0x7f25a80c8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T12:00:06.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.256+0000 7f25a77fe700 1 -- 172.21.15.78:0/3880671363 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a80c7c50 con 0x7f25a80a70c0 2023-10-14T12:00:06.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.257+0000 7f25a77fe700 1 --2- 172.21.15.78:0/3880671363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a80a70c0 0x7f25a8154960 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f25980098f0 tx=0x7f2598009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9606d7965e4eba70 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:06.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.257+0000 7f25a6ffd700 1 -- 172.21.15.78:0/3880671363 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2598004230 con 0x7f25a80a70c0 2023-10-14T12:00:06.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.257+0000 7f25a6ffd700 1 -- 172.21.15.78:0/3880671363 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2598004390 con 0x7f25a80a70c0 2023-10-14T12:00:06.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.257+0000 7f25a6ffd700 1 -- 172.21.15.78:0/3880671363 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25980036a0 con 0x7f25a80a70c0 2023-10-14T12:00:06.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.257+0000 7f25ae523700 1 -- 172.21.15.78:0/3880671363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a80a70c0 msgr2=0x7f25a8154960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:06.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.257+0000 7f25ae523700 1 --2- 172.21.15.78:0/3880671363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a80a70c0 0x7f25a8154960 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f25980098f0 tx=0x7f2598009c00 comp rx=0 tx=0).stop 2023-10-14T12:00:06.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.257+0000 7f25ae523700 1 -- 172.21.15.78:0/3880671363 shutdown_connections 2023-10-14T12:00:06.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.257+0000 7f25ae523700 1 --2- 172.21.15.78:0/3880671363 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f25a8154ea0 0x7f25a815d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:06.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.257+0000 7f25ae523700 1 --2- 172.21.15.78:0/3880671363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a80a70c0 0x7f25a8154960 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:06.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.257+0000 7f25ae523700 1 -- 172.21.15.78:0/3880671363 >> 172.21.15.78:0/3880671363 conn(0x7f25a80737d0 msgr2=0x7f25a8073be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:06.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.257+0000 7f25ae523700 1 -- 172.21.15.78:0/3880671363 shutdown_connections 2023-10-14T12:00:06.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.257+0000 7f25ae523700 1 -- 172.21.15.78:0/3880671363 wait complete. 2023-10-14T12:00:06.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.259+0000 7f25ae523700 1 Processor -- start 2023-10-14T12:00:06.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.259+0000 7f25ae523700 1 -- start start 2023-10-14T12:00:06.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.260+0000 7f25ae523700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f25a80a70c0 0x7f25a80bced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:06.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.260+0000 7f25ae523700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a81533d0 0x7f25a80ad440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:06.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.260+0000 7f25ae523700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f25a8154ea0 0x7f25a80c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:06.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.260+0000 7f25ae523700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a80c7a70 con 0x7f25a81533d0 2023-10-14T12:00:06.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.260+0000 7f25ae523700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a80bbf30 con 0x7f25a8154ea0 2023-10-14T12:00:06.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.260+0000 7f25ae523700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25a80bc210 con 0x7f25a80a70c0 2023-10-14T12:00:06.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.260+0000 7f25a77fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a81533d0 0x7f25a80ad440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:06.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.260+0000 7f25a77fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a81533d0 0x7f25a80ad440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47418/0 (socket says 172.21.15.78:47418) 2023-10-14T12:00:06.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.260+0000 7f25a77fe700 1 -- 172.21.15.78:0/4282467299 learned_addr learned my addr 172.21.15.78:0/4282467299 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:06.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.260+0000 7f25a7fff700 1 --2- 172.21.15.78:0/4282467299 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f25a80a70c0 0x7f25a80bced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:06.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.260+0000 7f25acac0700 1 --2- 172.21.15.78:0/4282467299 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f25a8154ea0 0x7f25a80c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:06.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.260+0000 7f25a77fe700 1 -- 172.21.15.78:0/4282467299 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f25a80a70c0 msgr2=0x7f25a80bced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:06.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.260+0000 7f25a77fe700 1 --2- 172.21.15.78:0/4282467299 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f25a80a70c0 0x7f25a80bced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:06.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.260+0000 7f25a77fe700 1 -- 172.21.15.78:0/4282467299 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f25a8154ea0 msgr2=0x7f25a80c43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:06.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.260+0000 7f25a77fe700 1 --2- 172.21.15.78:0/4282467299 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f25a8154ea0 0x7f25a80c43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:06.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.260+0000 7f25a77fe700 1 -- 172.21.15.78:0/4282467299 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2598009550 con 0x7f25a81533d0 2023-10-14T12:00:06.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.260+0000 7f25a77fe700 1 --2- 172.21.15.78:0/4282467299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a81533d0 0x7f25a80ad440 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f2598009130 tx=0x7f2598004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:06.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.260+0000 7f25a4ff9700 1 -- 172.21.15.78:0/4282467299 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2598003e30 con 0x7f25a81533d0 2023-10-14T12:00:06.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.260+0000 7f25a4ff9700 1 -- 172.21.15.78:0/4282467299 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f259801f070 con 0x7f25a81533d0 2023-10-14T12:00:06.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.260+0000 7f25ae523700 1 -- 172.21.15.78:0/4282467299 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25a80bec20 con 0x7f25a81533d0 2023-10-14T12:00:06.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.260+0000 7f25a4ff9700 1 -- 172.21.15.78:0/4282467299 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259800b8a0 con 0x7f25a81533d0 2023-10-14T12:00:06.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.261+0000 7f25ae523700 1 -- 172.21.15.78:0/4282467299 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25a80be8d0 con 0x7f25a81533d0 2023-10-14T12:00:06.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.262+0000 7f25ae523700 1 -- 172.21.15.78:0/4282467299 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25a80bcbf0 con 0x7f25a81533d0 2023-10-14T12:00:06.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.262+0000 7f25a4ff9700 1 -- 172.21.15.78:0/4282467299 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2598021030 con 0x7f25a81533d0 2023-10-14T12:00:06.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.262+0000 7f25a4ff9700 1 --2- 172.21.15.78:0/4282467299 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f259406af00 0x7f259406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:06.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.262+0000 7f25a4ff9700 1 -- 172.21.15.78:0/4282467299 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f259808a990 con 0x7f25a81533d0 2023-10-14T12:00:06.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.262+0000 7f25a7fff700 1 --2- 172.21.15.78:0/4282467299 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f259406af00 0x7f259406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:06.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.263+0000 7f25a7fff700 1 --2- 172.21.15.78:0/4282467299 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f259406af00 0x7f259406d3c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f259c004f30 tx=0x7f259c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:06.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.267+0000 7f25a4ff9700 1 -- 172.21.15.78:0/4282467299 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f25a80bcbf0 con 0x7f25a81533d0 2023-10-14T12:00:06.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.454+0000 7f25ae523700 1 -- 172.21.15.78:0/4282467299 --> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f25a80bc380 con 0x7f259406af00 2023-10-14T12:00:06.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.454+0000 7f25a4ff9700 1 -- 172.21.15.78:0/4282467299 <== mgr.4104 v2:172.21.15.78:6832/3171890074 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f25a80bc380 con 0x7f259406af00 2023-10-14T12:00:06.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.456+0000 7f25ae523700 1 -- 172.21.15.78:0/4282467299 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f259406af00 msgr2=0x7f259406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:06.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.456+0000 7f25ae523700 1 --2- 172.21.15.78:0/4282467299 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f259406af00 0x7f259406d3c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f259c004f30 tx=0x7f259c009040 comp rx=0 tx=0).stop 2023-10-14T12:00:06.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.456+0000 7f25ae523700 1 -- 172.21.15.78:0/4282467299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a81533d0 msgr2=0x7f25a80ad440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:06.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.456+0000 7f25ae523700 1 --2- 172.21.15.78:0/4282467299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a81533d0 0x7f25a80ad440 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f2598009130 tx=0x7f2598004750 comp rx=0 tx=0).stop 2023-10-14T12:00:06.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.456+0000 7f25ae523700 1 -- 172.21.15.78:0/4282467299 shutdown_connections 2023-10-14T12:00:06.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.456+0000 7f25ae523700 1 --2- 172.21.15.78:0/4282467299 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f259406af00 0x7f259406d3c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:06.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.456+0000 7f25ae523700 1 --2- 172.21.15.78:0/4282467299 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f25a8154ea0 0x7f25a80c43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:06.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.456+0000 7f25ae523700 1 --2- 172.21.15.78:0/4282467299 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f25a80a70c0 0x7f25a80bced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:06.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.456+0000 7f25ae523700 1 --2- 172.21.15.78:0/4282467299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a81533d0 0x7f25a80ad440 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:06.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.456+0000 7f25ae523700 1 -- 172.21.15.78:0/4282467299 >> 172.21.15.78:0/4282467299 conn(0x7f25a80737d0 msgr2=0x7f25a80c85d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:06.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.456+0000 7f25ae523700 1 -- 172.21.15.78:0/4282467299 shutdown_connections 2023-10-14T12:00:06.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.456+0000 7f25ae523700 1 -- 172.21.15.78:0/4282467299 wait complete. 2023-10-14T12:00:06.462 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 7 to deep-scrub 2023-10-14T12:00:06.470 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T12:00:06.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.587+0000 7f6ad777e700 1 Processor -- start 2023-10-14T12:00:06.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.587+0000 7f6ad777e700 1 -- start start 2023-10-14T12:00:06.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.588+0000 7f6ad777e700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6ad00c33d0 0x7f6ad0150f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:06.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.588+0000 7f6ad777e700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ad00c11a0 0x7f6ad006e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:06.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.588+0000 7f6ad777e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad0070ff0 con 0x7f6ad00a7ba0 2023-10-14T12:00:06.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.588+0000 7f6ad777e700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ad0154f90 con 0x7f6ad00c33d0 2023-10-14T12:00:06.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.588+0000 7f6ad777e700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad0155270 con 0x7f6ad00c11a0 2023-10-14T12:00:06.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.588+0000 7f6ad551a700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6ad00c33d0 0x7f6ad0150f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:06.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.588+0000 7f6ad551a700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6ad00c33d0 0x7f6ad0150f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47424/0 (socket says 172.21.15.78:47424) 2023-10-14T12:00:06.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.588+0000 7f6ad551a700 1 -- 172.21.15.78:0/128333861 learned_addr learned my addr 172.21.15.78:0/128333861 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:06.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.588+0000 7f6ad551a700 1 -- 172.21.15.78:0/128333861 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ad00c11a0 msgr2=0x7f6ad006e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T12:00:06.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.588+0000 7f6ad551a700 1 --2- 172.21.15.78:0/128333861 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ad00c11a0 0x7f6ad006e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:06.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.588+0000 7f6ad551a700 1 -- 172.21.15.78:0/128333861 >> v1:172.21.15.195:6789/0 conn(0x7f6ad00a7ba0 legacy=0x7f6ad014e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T12:00:06.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.588+0000 7f6ad551a700 1 -- 172.21.15.78:0/128333861 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad0070c80 con 0x7f6ad00c33d0 2023-10-14T12:00:06.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.589+0000 7f6ad551a700 1 --2- 172.21.15.78:0/128333861 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ad00c33d0 0x7f6ad0150f70 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f6ac00098f0 tx=0x7f6ac0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=966c23b6518b1c18 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:06.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.589+0000 7f6ac7fff700 1 -- 172.21.15.78:0/128333861 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac0004230 con 0x7f6ad00c33d0 2023-10-14T12:00:06.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.589+0000 7f6ac7fff700 1 -- 172.21.15.78:0/128333861 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ac0004390 con 0x7f6ad00c33d0 2023-10-14T12:00:06.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.589+0000 7f6ac7fff700 1 -- 172.21.15.78:0/128333861 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac00036a0 con 0x7f6ad00c33d0 2023-10-14T12:00:06.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.589+0000 7f6ad777e700 1 -- 172.21.15.78:0/128333861 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ad00c33d0 msgr2=0x7f6ad0150f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:06.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.589+0000 7f6ad777e700 1 --2- 172.21.15.78:0/128333861 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ad00c33d0 0x7f6ad0150f70 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f6ac00098f0 tx=0x7f6ac0009c00 comp rx=0 tx=0).stop 2023-10-14T12:00:06.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.589+0000 7f6ad777e700 1 -- 172.21.15.78:0/128333861 shutdown_connections 2023-10-14T12:00:06.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.589+0000 7f6ad777e700 1 --2- 172.21.15.78:0/128333861 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ad00c33d0 0x7f6ad0150f70 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:06.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.589+0000 7f6ad777e700 1 --2- 172.21.15.78:0/128333861 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ad00c11a0 0x7f6ad006e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:06.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.589+0000 7f6ad777e700 1 -- 172.21.15.78:0/128333861 >> 172.21.15.78:0/128333861 conn(0x7f6ad00a8410 msgr2=0x7f6ad00a8820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:06.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.589+0000 7f6ad777e700 1 -- 172.21.15.78:0/128333861 shutdown_connections 2023-10-14T12:00:06.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.589+0000 7f6ad777e700 1 -- 172.21.15.78:0/128333861 wait complete. 2023-10-14T12:00:06.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.591+0000 7f6ad777e700 1 Processor -- start 2023-10-14T12:00:06.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.591+0000 7f6ad777e700 1 -- start start 2023-10-14T12:00:06.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.591+0000 7f6ad777e700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ad00a7ba0 0x7f6ad00c2500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:06.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.591+0000 7f6ad777e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ad00c11a0 0x7f6ad00b1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:06.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.591+0000 7f6ad777e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ad00c33d0 0x7f6ad00b2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:06.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.591+0000 7f6ad777e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad00a7240 con 0x7f6ad00c33d0 2023-10-14T12:00:06.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.591+0000 7f6ad777e700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad0154b50 con 0x7f6ad00c11a0 2023-10-14T12:00:06.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.591+0000 7f6ad777e700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad00a6fa0 con 0x7f6ad00a7ba0 2023-10-14T12:00:06.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.591+0000 7f6ad5d1b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ad00c33d0 0x7f6ad00b2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:06.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.591+0000 7f6ad5d1b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ad00c33d0 0x7f6ad00b2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47434/0 (socket says 172.21.15.78:47434) 2023-10-14T12:00:06.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.591+0000 7f6ad5d1b700 1 -- 172.21.15.78:0/3985312352 learned_addr learned my addr 172.21.15.78:0/3985312352 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:06.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.592+0000 7f6ad551a700 1 --2- 172.21.15.78:0/3985312352 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ad00a7ba0 0x7f6ad00c2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:06.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.592+0000 7f6ad4d19700 1 --2- 172.21.15.78:0/3985312352 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ad00c11a0 0x7f6ad00b1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:06.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.592+0000 7f6ad5d1b700 1 -- 172.21.15.78:0/3985312352 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ad00a7ba0 msgr2=0x7f6ad00c2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:06.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.592+0000 7f6ad5d1b700 1 --2- 172.21.15.78:0/3985312352 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ad00a7ba0 0x7f6ad00c2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:06.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.592+0000 7f6ad5d1b700 1 -- 172.21.15.78:0/3985312352 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ad00c11a0 msgr2=0x7f6ad00b1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:06.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.592+0000 7f6ad5d1b700 1 --2- 172.21.15.78:0/3985312352 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ad00c11a0 0x7f6ad00b1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:06.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.592+0000 7f6ad5d1b700 1 -- 172.21.15.78:0/3985312352 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac0009550 con 0x7f6ad00c33d0 2023-10-14T12:00:06.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.592+0000 7f6ad5d1b700 1 --2- 172.21.15.78:0/3985312352 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ad00c33d0 0x7f6ad00b2800 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f6ac800c7a0 tx=0x7f6ac800cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:06.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.592+0000 7f6ac5ffb700 1 -- 172.21.15.78:0/3985312352 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac8003e10 con 0x7f6ad00c33d0 2023-10-14T12:00:06.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.592+0000 7f6ac5ffb700 1 -- 172.21.15.78:0/3985312352 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ac8005070 con 0x7f6ad00c33d0 2023-10-14T12:00:06.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.592+0000 7f6ad777e700 1 -- 172.21.15.78:0/3985312352 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ad014edf0 con 0x7f6ad00c33d0 2023-10-14T12:00:06.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.592+0000 7f6ac5ffb700 1 -- 172.21.15.78:0/3985312352 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac8005420 con 0x7f6ad00c33d0 2023-10-14T12:00:06.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.592+0000 7f6ad777e700 1 -- 172.21.15.78:0/3985312352 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ad01507e0 con 0x7f6ad00c33d0 2023-10-14T12:00:06.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.593+0000 7f6ad777e700 1 -- 172.21.15.78:0/3985312352 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ad0074b80 con 0x7f6ad00c33d0 2023-10-14T12:00:06.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.594+0000 7f6ac5ffb700 1 -- 172.21.15.78:0/3985312352 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6ac8005580 con 0x7f6ad00c33d0 2023-10-14T12:00:06.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.594+0000 7f6ac5ffb700 1 --2- 172.21.15.78:0/3985312352 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f6ab406af00 0x7f6ab406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:06.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.594+0000 7f6ac5ffb700 1 -- 172.21.15.78:0/3985312352 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f6ac8088980 con 0x7f6ad00c33d0 2023-10-14T12:00:06.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.594+0000 7f6ad551a700 1 --2- 172.21.15.78:0/3985312352 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f6ab406af00 0x7f6ab406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:06.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.594+0000 7f6ad551a700 1 --2- 172.21.15.78:0/3985312352 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f6ab406af00 0x7f6ab406d3c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6ac0009130 tx=0x7f6ac0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:06.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.598+0000 7f6ac5ffb700 1 -- 172.21.15.78:0/3985312352 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6ac8053dc0 con 0x7f6ad00c33d0 2023-10-14T12:00:06.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.784+0000 7f6ad777e700 1 -- 172.21.15.78:0/3985312352 --> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6ad014ec70 con 0x7f6ab406af00 2023-10-14T12:00:06.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.786+0000 7f6ac5ffb700 1 -- 172.21.15.78:0/3985312352 <== mgr.4104 v2:172.21.15.78:6832/3171890074 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+51153 (secure 0 0 0) 0x7f6ac0020070 con 0x7f6ab406af00 2023-10-14T12:00:06.789 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T12:00:06.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.788+0000 7f6ad777e700 1 -- 172.21.15.78:0/3985312352 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f6ab406af00 msgr2=0x7f6ab406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:06.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.788+0000 7f6ad777e700 1 --2- 172.21.15.78:0/3985312352 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f6ab406af00 0x7f6ab406d3c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6ac0009130 tx=0x7f6ac0008030 comp rx=0 tx=0).stop 2023-10-14T12:00:06.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.788+0000 7f6ad777e700 1 -- 172.21.15.78:0/3985312352 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ad00c33d0 msgr2=0x7f6ad00b2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:06.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.788+0000 7f6ad777e700 1 --2- 172.21.15.78:0/3985312352 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ad00c33d0 0x7f6ad00b2800 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f6ac800c7a0 tx=0x7f6ac800cb60 comp rx=0 tx=0).stop 2023-10-14T12:00:06.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.789+0000 7f6ad777e700 1 -- 172.21.15.78:0/3985312352 shutdown_connections 2023-10-14T12:00:06.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.789+0000 7f6ad777e700 1 --2- 172.21.15.78:0/3985312352 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ad00c33d0 0x7f6ad00b2800 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:06.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.789+0000 7f6ad777e700 1 --2- 172.21.15.78:0/3985312352 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ad00a7ba0 0x7f6ad00c2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:06.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.789+0000 7f6ad777e700 1 --2- 172.21.15.78:0/3985312352 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f6ab406af00 0x7f6ab406d3c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:06.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.789+0000 7f6ad777e700 1 --2- 172.21.15.78:0/3985312352 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ad00c11a0 0x7f6ad00b1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:06.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.789+0000 7f6ad777e700 1 -- 172.21.15.78:0/3985312352 >> 172.21.15.78:0/3985312352 conn(0x7f6ad00a8410 msgr2=0x7f6ad0144140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:06.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.789+0000 7f6ad777e700 1 -- 172.21.15.78:0/3985312352 shutdown_connections 2023-10-14T12:00:06.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:06.789+0000 7f6ad777e700 1 -- 172.21.15.78:0/3985312352 wait complete. 2023-10-14T12:00:06.794 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2023-10-14T12:00:06.802 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2023-10-14T12:00:04.990050+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":219472,"kb_used_data":3296,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755248,"statfs":{"total":773094113280,"available":772869373952,"internally_reserved":0,"allocated":3375104,"data_stored":2433056,"data_compressed":65244,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.485791"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":176,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T11:59:56.504026+0000","last_change":"2023-10-14T11:58:10.616726+0000","last_active":"2023-10-14T11:59:56.504026+0000","last_peered":"2023-10-14T11:59:56.504026+0000","last_clean":"2023-10-14T11:59:56.504026+0000","last_became_active":"2023-10-14T11:58:10.615628+0000","last_became_peered":"2023-10-14T11:58:10.615628+0000","last_unstale":"2023-10-14T11:59:56.504026+0000","last_undegraded":"2023-10-14T11:59:56.504026+0000","last_fullsized":"2023-10-14T11:59:56.504026+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T11:58:09.001265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T11:58:09.001265+0000","last_clean_scrub_stamp":"2023-10-14T11:58:09.001265+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:22:50.496050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65244,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27480,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344360,"statfs":{"total":96636764160,"available":96608624640,"internally_reserved":0,"allocated":471040,"data_stored":746908,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.496},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.63100000000000001}]},{"osd":1,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.58499999999999996}]},{"osd":2,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":2.8250000000000002,"5min":2.8250000000000002,"15min":2.8250000000000002},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.61199999999999999}]},{"osd":3,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":2.8639999999999999,"5min":2.8639999999999999,"15min":2.8639999999999999},"last":0.64400000000000002}]},{"osd":4,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.55000000000000004}]},{"osd":5,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.46999999999999997}]},{"osd":6,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.59799999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574874,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27416,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":405504,"data_stored":156540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.58599999999999997}]},{"osd":1,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.60999999999999999}]},{"osd":3,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":2.2789999999999999,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.64600000000000002}]},{"osd":4,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.73199999999999998}]},{"osd":5,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.626}]},{"osd":7,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.78100000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27420,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":405504,"data_stored":156540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.374},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.32900000000000001}]},{"osd":2,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.42599999999999999}]},{"osd":3,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.495,"5min":0.495,"15min":0.495},"last":0.36499999999999999}]},{"osd":4,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.56999999999999995}]},{"osd":5,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.67200000000000004}]},{"osd":6,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.65600000000000003}]},{"osd":7,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.54200000000000004}]}]},{"osd":0,"up_from":13,"seq":55834574873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27484,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344356,"statfs":{"total":96636764160,"available":96608620544,"internally_reserved":0,"allocated":471040,"data_stored":746908,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.31,"15min":0.31},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"last":0.35199999999999998}]},{"osd":2,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.378,"5min":0.378,"15min":0.378},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.31900000000000001}]},{"osd":3,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.439,"5min":0.439,"15min":0.439},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.36199999999999999}]},{"osd":4,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":2.3839999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":2.4100000000000001}]},{"osd":5,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.45400000000000001}]},{"osd":6,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.69699999999999995}]},{"osd":7,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.57699999999999996}]}]},{"osd":2,"up_from":13,"seq":55834574873,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27420,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":405504,"data_stored":156540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 11:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.38500000000000001}]},{"osd":1,"last update":"Sat Oct 14 11:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.32500000000000001}]},{"osd":3,"last update":"Sat Oct 14 11:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.46100000000000002}]},{"osd":4,"last update":"Sat Oct 14 11:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1,"5min":1,"15min":1},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.83799999999999997}]},{"osd":5,"last update":"Sat Oct 14 11:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.80400000000000005}]},{"osd":6,"last update":"Sat Oct 14 11:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.68600000000000005}]},{"osd":7,"last update":"Sat Oct 14 11:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.73199999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27416,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":405504,"data_stored":156540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 11:59:10 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.226,"5min":7.226,"15min":7.226},"last":0.38100000000000001},{"interface":"front","average":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":21.067,"5min":21.067,"15min":21.067},"last":0.30599999999999999}]},{"osd":1,"last update":"Sat Oct 14 11:59:10 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":7.3419999999999996,"5min":7.3419999999999996,"15min":7.3419999999999996},"last":0.33400000000000002}]},{"osd":2,"last update":"Sat Oct 14 11:59:10 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":7.3120000000000003,"5min":7.3120000000000003,"15min":7.3120000000000003},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":7.1870000000000003,"5min":7.1870000000000003,"15min":7.1870000000000003},"last":0.40899999999999997}]},{"osd":4,"last update":"Sat Oct 14 11:59:10 2023","interfaces":[{"interface":"back","average":{"1min":1.8640000000000001,"5min":1.8640000000000001,"15min":1.8640000000000001},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":20.984000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.52800000000000002},{"interface":"front","average":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":17.902999999999999,"5min":17.902999999999999,"15min":17.902999999999999},"last":0.48599999999999999}]},{"osd":5,"last update":"Sat Oct 14 11:59:10 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":7.2560000000000002,"5min":7.2560000000000002,"15min":7.2560000000000002},"last":0.57599999999999996},{"interface":"front","average":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":21.100000000000001,"5min":21.100000000000001,"15min":21.100000000000001},"last":0.61699999999999999}]},{"osd":6,"last update":"Sat Oct 14 11:59:10 2023","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.002,"15min":1.002},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":7.1589999999999998,"5min":7.1589999999999998,"15min":7.1589999999999998},"last":0.51600000000000001},{"interface":"front","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"last":0.47299999999999998}]},{"osd":7,"last update":"Sat Oct 14 11:59:10 2023","interfaces":[{"interface":"back","average":{"1min":1.891,"5min":1.891,"15min":1.891},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":21.035,"5min":21.035,"15min":21.035},"last":0.498},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":7.1230000000000002,"5min":7.1230000000000002,"15min":7.1230000000000002},"last":0.56599999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27416,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":405504,"data_stored":156540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":4.6159999999999997,"5min":4.6159999999999997,"15min":4.6159999999999997},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.57899999999999996}]},{"osd":1,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":3.6600000000000001,"5min":3.6600000000000001,"15min":3.6600000000000001},"last":0.56699999999999995}]},{"osd":2,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":5.7160000000000002,"5min":5.7160000000000002,"15min":5.7160000000000002},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.60799999999999998}]},{"osd":3,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":3.7549999999999999,"5min":3.7549999999999999,"15min":3.7549999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.744,"5min":1.744,"15min":1.744},"last":0.624}]},{"osd":5,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.372},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.48799999999999999}]},{"osd":6,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.46999999999999997}]},{"osd":7,"last update":"Sat Oct 14 11:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.55000000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27420,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":405504,"data_stored":156540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.68999999999999995}]},{"osd":1,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.68000000000000005}]},{"osd":2,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.74099999999999999}]},{"osd":3,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.77600000000000002}]},{"osd":4,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.64300000000000002}]},{"osd":6,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.70199999999999996}]},{"osd":7,"last update":"Sat Oct 14 11:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.754},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1,"5min":1,"15min":1},"last":0.72899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T12:00:06.806 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-14T11:58:09.001265+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=11, tm_min=58, tm_sec=9, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=12, tm_min=0, tm_sec=1, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T12:00:06.806 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T12:00:23.745 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T12:00:23.750 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T12:00:26.808 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T12:00:26.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.940+0000 7f0b05786700 1 Processor -- start 2023-10-14T12:00:26.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.940+0000 7f0b05786700 1 -- start start 2023-10-14T12:00:26.944 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.940+0000 7f0b05786700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0b0006df50 0x7f0b000a9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:26.945 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.940+0000 7f0b05786700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b0006c970 con 0x7f0b0014df40 2023-10-14T12:00:26.945 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.940+0000 7f0b05786700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b0006dad0 con 0x7f0b001517c0 2023-10-14T12:00:26.945 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.940+0000 7f0b05786700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b0006cfa0 con 0x7f0b0006df50 2023-10-14T12:00:26.946 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.941+0000 7f0afeffd700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0b0006df50 0x7f0b000a9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:26.946 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.941+0000 7f0aff7fe700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f0b0014df40 0x7f0b0015b6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52318/0 (socket says 172.21.15.78:52318) 2023-10-14T12:00:26.946 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.941+0000 7f0afeffd700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0b0006df50 0x7f0b000a9660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47438/0 (socket says 172.21.15.78:47438) 2023-10-14T12:00:26.946 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.941+0000 7f0aff7fe700 1 -- 172.21.15.78:0/3298804150 learned_addr learned my addr 172.21.15.78:0/3298804150 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T12:00:26.947 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.941+0000 7f0afeffd700 1 -- 172.21.15.78:0/3298804150 >> v1:172.21.15.195:6789/0 conn(0x7f0b001517c0 legacy=0x7f0b0006d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:26.947 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.941+0000 7f0afeffd700 1 -- 172.21.15.78:0/3298804150 >> v1:172.21.15.78:6789/0 conn(0x7f0b0014df40 legacy=0x7f0b0015b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:26.947 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.941+0000 7f0afdffb700 1 -- 172.21.15.78:0/3298804150 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3474351207 0 0) 0x7f0b0006c970 con 0x7f0b0014df40 2023-10-14T12:00:26.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.941+0000 7f0afeffd700 1 -- 172.21.15.78:0/3298804150 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b000abb80 con 0x7f0b0006df50 2023-10-14T12:00:26.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.941+0000 7f0afeffd700 1 --2- 172.21.15.78:0/3298804150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b0006df50 0x7f0b000a9660 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f0af000a9e0 tx=0x7f0af000acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b766c34599a3cbe7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:26.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.941+0000 7f0afdffb700 1 -- 172.21.15.78:0/3298804150 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af000cb70 con 0x7f0b0006df50 2023-10-14T12:00:26.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.941+0000 7f0afdffb700 1 -- 172.21.15.78:0/3298804150 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0af000ccd0 con 0x7f0b0006df50 2023-10-14T12:00:26.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.941+0000 7f0afdffb700 1 -- 172.21.15.78:0/3298804150 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af0018620 con 0x7f0b0006df50 2023-10-14T12:00:26.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.942+0000 7f0b05786700 1 -- 172.21.15.78:0/3298804150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b0006df50 msgr2=0x7f0b000a9660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:26.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.942+0000 7f0b05786700 1 --2- 172.21.15.78:0/3298804150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b0006df50 0x7f0b000a9660 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f0af000a9e0 tx=0x7f0af000acf0 comp rx=0 tx=0).stop 2023-10-14T12:00:26.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.942+0000 7f0b05786700 1 -- 172.21.15.78:0/3298804150 shutdown_connections 2023-10-14T12:00:26.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.942+0000 7f0b05786700 1 --2- 172.21.15.78:0/3298804150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b0006df50 0x7f0b000a9660 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:26.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.942+0000 7f0b05786700 1 -- 172.21.15.78:0/3298804150 >> 172.21.15.78:0/3298804150 conn(0x7f0b000c2930 msgr2=0x7f0b000c2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:26.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.942+0000 7f0b05786700 1 -- 172.21.15.78:0/3298804150 shutdown_connections 2023-10-14T12:00:26.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.942+0000 7f0b05786700 1 -- 172.21.15.78:0/3298804150 wait complete. 2023-10-14T12:00:26.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.943+0000 7f0b05786700 1 Processor -- start 2023-10-14T12:00:26.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.943+0000 7f0b05786700 1 -- start start 2023-10-14T12:00:26.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.944+0000 7f0b05786700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0b0006df50 0x7f0b00147390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:26.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.944+0000 7f0b05786700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b0014df40 0x7f0b00143d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:26.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.944+0000 7f0b05786700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b001517c0 0x7f0b001442b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:26.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.944+0000 7f0b05786700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b00071290 con 0x7f0b001517c0 2023-10-14T12:00:26.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.944+0000 7f0b05786700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b001447f0 con 0x7f0b0014df40 2023-10-14T12:00:26.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.944+0000 7f0b05786700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b000721c0 con 0x7f0b0006df50 2023-10-14T12:00:26.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.944+0000 7f0aff7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b001517c0 0x7f0b001442b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:26.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.944+0000 7f0aff7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b001517c0 0x7f0b001442b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47448/0 (socket says 172.21.15.78:47448) 2023-10-14T12:00:26.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.944+0000 7f0aff7fe700 1 -- 172.21.15.78:0/2855778440 learned_addr learned my addr 172.21.15.78:0/2855778440 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:26.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.944+0000 7f0afeffd700 1 --2- 172.21.15.78:0/2855778440 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0b0006df50 0x7f0b00147390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:26.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.944+0000 7f0afe7fc700 1 --2- 172.21.15.78:0/2855778440 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b0014df40 0x7f0b00143d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:26.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.944+0000 7f0aff7fe700 1 -- 172.21.15.78:0/2855778440 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0b0006df50 msgr2=0x7f0b00147390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:26.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.944+0000 7f0aff7fe700 1 --2- 172.21.15.78:0/2855778440 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0b0006df50 0x7f0b00147390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:26.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.944+0000 7f0aff7fe700 1 -- 172.21.15.78:0/2855778440 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b0014df40 msgr2=0x7f0b00143d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:26.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.944+0000 7f0aff7fe700 1 --2- 172.21.15.78:0/2855778440 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b0014df40 0x7f0b00143d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:26.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.944+0000 7f0aff7fe700 1 -- 172.21.15.78:0/2855778440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0af000a640 con 0x7f0b001517c0 2023-10-14T12:00:26.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.944+0000 7f0aff7fe700 1 --2- 172.21.15.78:0/2855778440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b001517c0 0x7f0b001442b0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f0af800cb60 tx=0x7f0af800cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:26.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.945+0000 7f0ae77fe700 1 -- 172.21.15.78:0/2855778440 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af800ab50 con 0x7f0b001517c0 2023-10-14T12:00:26.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.945+0000 7f0ae77fe700 1 -- 172.21.15.78:0/2855778440 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0af800acb0 con 0x7f0b001517c0 2023-10-14T12:00:26.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.945+0000 7f0b05786700 1 -- 172.21.15.78:0/2855778440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b0014c3d0 con 0x7f0b001517c0 2023-10-14T12:00:26.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.945+0000 7f0ae77fe700 1 -- 172.21.15.78:0/2855778440 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af8007310 con 0x7f0b001517c0 2023-10-14T12:00:26.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.945+0000 7f0b05786700 1 -- 172.21.15.78:0/2855778440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b0014c6b0 con 0x7f0b001517c0 2023-10-14T12:00:26.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.946+0000 7f0b05786700 1 -- 172.21.15.78:0/2855778440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b00074b80 con 0x7f0b001517c0 2023-10-14T12:00:26.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.946+0000 7f0ae77fe700 1 -- 172.21.15.78:0/2855778440 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0af8007470 con 0x7f0b001517c0 2023-10-14T12:00:26.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.946+0000 7f0ae77fe700 1 --2- 172.21.15.78:0/2855778440 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f0aec06af00 0x7f0aec06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:26.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.947+0000 7f0ae77fe700 1 -- 172.21.15.78:0/2855778440 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0af8089100 con 0x7f0b001517c0 2023-10-14T12:00:26.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.947+0000 7f0afeffd700 1 --2- 172.21.15.78:0/2855778440 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f0aec06af00 0x7f0aec06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:26.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.947+0000 7f0afeffd700 1 --2- 172.21.15.78:0/2855778440 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f0aec06af00 0x7f0aec06d3c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0af000a1c0 tx=0x7f0af0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:26.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:26.951+0000 7f0ae77fe700 1 -- 172.21.15.78:0/2855778440 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0af80544a0 con 0x7f0b001517c0 2023-10-14T12:00:27.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.135+0000 7f0b05786700 1 -- 172.21.15.78:0/2855778440 --> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0b0006cbc0 con 0x7f0aec06af00 2023-10-14T12:00:27.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.138+0000 7f0ae77fe700 1 -- 172.21.15.78:0/2855778440 <== mgr.4104 v2:172.21.15.78:6832/3171890074 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+51010 (secure 0 0 0) 0x7f0af001d040 con 0x7f0aec06af00 2023-10-14T12:00:27.140 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T12:00:27.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.140+0000 7f0b05786700 1 -- 172.21.15.78:0/2855778440 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f0aec06af00 msgr2=0x7f0aec06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:27.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.140+0000 7f0b05786700 1 --2- 172.21.15.78:0/2855778440 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f0aec06af00 0x7f0aec06d3c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0af000a1c0 tx=0x7f0af0008040 comp rx=0 tx=0).stop 2023-10-14T12:00:27.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.140+0000 7f0b05786700 1 -- 172.21.15.78:0/2855778440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b001517c0 msgr2=0x7f0b001442b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:27.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.140+0000 7f0b05786700 1 --2- 172.21.15.78:0/2855778440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b001517c0 0x7f0b001442b0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f0af800cb60 tx=0x7f0af800cf20 comp rx=0 tx=0).stop 2023-10-14T12:00:27.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.140+0000 7f0b05786700 1 -- 172.21.15.78:0/2855778440 shutdown_connections 2023-10-14T12:00:27.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.140+0000 7f0b05786700 1 --2- 172.21.15.78:0/2855778440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b001517c0 0x7f0b001442b0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:27.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.140+0000 7f0b05786700 1 --2- 172.21.15.78:0/2855778440 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0b0006df50 0x7f0b00147390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:27.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.140+0000 7f0b05786700 1 --2- 172.21.15.78:0/2855778440 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7f0aec06af00 0x7f0aec06d3c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:27.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.140+0000 7f0b05786700 1 --2- 172.21.15.78:0/2855778440 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b0014df40 0x7f0b00143d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:27.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.140+0000 7f0b05786700 1 -- 172.21.15.78:0/2855778440 >> 172.21.15.78:0/2855778440 conn(0x7f0b000c2930 msgr2=0x7f0b000c4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:27.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.140+0000 7f0b05786700 1 -- 172.21.15.78:0/2855778440 shutdown_connections 2023-10-14T12:00:27.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.140+0000 7f0b05786700 1 -- 172.21.15.78:0/2855778440 wait complete. 2023-10-14T12:00:27.146 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2023-10-14T12:00:27.154 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2023-10-14T12:00:26.994534+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":219460,"kb_used_data":3296,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755260,"statfs":{"total":773094113280,"available":772869386240,"internally_reserved":0,"allocated":3375104,"data_stored":2433056,"data_compressed":65244,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002423"},"pg_stats":[{"pgid":"1.0","version":"35'119","reported_seq":188,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-14T12:00:10.935432+0000","last_change":"2023-10-14T12:00:10.935432+0000","last_active":"2023-10-14T12:00:10.935432+0000","last_peered":"2023-10-14T12:00:10.935432+0000","last_clean":"2023-10-14T12:00:10.935432+0000","last_became_active":"2023-10-14T11:58:10.615628+0000","last_became_peered":"2023-10-14T11:58:10.615628+0000","last_unstale":"2023-10-14T12:00:10.935432+0000","last_undegraded":"2023-10-14T12:00:10.935432+0000","last_fullsized":"2023-10-14T12:00:10.935432+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'119","last_scrub_stamp":"2023-10-14T12:00:10.935232+0000","last_deep_scrub":"35'119","last_deep_scrub_stamp":"2023-10-14T12:00:10.935232+0000","last_clean_scrub_stamp":"2023-10-14T12:00:10.935232+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T13:29:27.725913+0000","scrub_duration":0.0052467850000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65244,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574878,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27476,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":471040,"data_stored":746908,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 12:00:13 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.55700000000000005,"15min":0.55100000000000005},"min":{"1min":0.42799999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.58499999999999996,"15min":0.58099999999999996},"min":{"1min":0.435,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.69399999999999995}]},{"osd":1,"last update":"Sat Oct 14 12:00:13 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.53800000000000003,"15min":0.53300000000000003},"min":{"1min":0.317,"5min":0.312,"15min":0.312},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.55100000000000005,"15min":0.55200000000000005},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.51000000000000001}]},{"osd":2,"last update":"Sat Oct 14 12:00:13 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.622,"15min":0.63100000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.74399999999999999,"5min":2.8250000000000002,"15min":2.8250000000000002},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.56599999999999995,"15min":0.56799999999999995},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.72299999999999998}]},{"osd":3,"last update":"Sat Oct 14 12:00:13 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.59699999999999998,"15min":0.59499999999999997},"min":{"1min":0.435,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.83099999999999996,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.69599999999999995,"15min":0.70599999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.79400000000000004,"5min":2.8639999999999999,"15min":2.8639999999999999},"last":0.70799999999999996}]},{"osd":4,"last update":"Sat Oct 14 12:00:13 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.52100000000000002,"15min":0.53300000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.755,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59899999999999998,"15min":0.60399999999999998},"min":{"1min":0.44600000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.73199999999999998,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.56299999999999994}]},{"osd":5,"last update":"Sat Oct 14 12:00:13 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.58099999999999996,"15min":0.59099999999999997},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.59399999999999997,"15min":0.60399999999999998},"min":{"1min":0.36799999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77400000000000002,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.66600000000000004}]},{"osd":6,"last update":"Sat Oct 14 12:00:13 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.61599999999999999,"15min":0.625},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.80200000000000005,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.64500000000000002,"15min":0.65500000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.81399999999999995,"5min":1.151,"15min":1.151},"last":0.75}]}]},{"osd":6,"up_from":13,"seq":55834574878,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27416,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":405504,"data_stored":156540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 12:00:11 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54800000000000004,"15min":0.55000000000000004},"min":{"1min":0.41299999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.71699999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.57699999999999996,"15min":0.58299999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.58999999999999997}]},{"osd":1,"last update":"Sat Oct 14 12:00:11 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54800000000000004,"15min":0.54900000000000004},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.72799999999999998,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.61799999999999999,"15min":0.627},"min":{"1min":0.42499999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.745,"5min":1.173,"15min":1.173},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Oct 14 12:00:11 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.60399999999999998,"15min":0.61199999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.73399999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.55900000000000005,"15min":0.56100000000000005},"min":{"1min":0.41999999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.46400000000000002}]},{"osd":3,"last update":"Sat Oct 14 12:00:11 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.69099999999999995,"15min":0.70799999999999996},"min":{"1min":0.46800000000000003,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.68000000000000005,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":0.44},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.61399999999999999,"15min":0.62},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.75600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.53900000000000003}]},{"osd":4,"last update":"Sat Oct 14 12:00:11 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.60199999999999998,"15min":0.61599999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71599999999999997,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.66900000000000004,"15min":0.69299999999999995},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73199999999999998,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.60999999999999999}]},{"osd":5,"last update":"Sat Oct 14 12:00:11 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.61699999999999999,"15min":0.628},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.78800000000000003,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.64000000000000001,"15min":0.65500000000000003},"min":{"1min":0.41099999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.71799999999999997,"5min":1.042,"15min":1.042},"last":0.51400000000000001}]},{"osd":7,"last update":"Sat Oct 14 12:00:11 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.67300000000000004,"15min":0.68700000000000006},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.82499999999999996,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.629},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.71799999999999997,"15min":0.73499999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.78100000000000003,"5min":1.25,"15min":1.25},"last":0.68500000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574878,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27420,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":405504,"data_stored":156540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 12:00:11 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34699999999999998,"15min":0.34799999999999998},"min":{"1min":0.23899999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.46000000000000002,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.375},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.33900000000000002,"15min":0.33100000000000002},"min":{"1min":0.28999999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52700000000000002,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.441}]},{"osd":2,"last update":"Sat Oct 14 12:00:11 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.32500000000000001,"15min":0.32100000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"last":0.505},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.38,"15min":0.375},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.60899999999999999,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.49299999999999999}]},{"osd":3,"last update":"Sat Oct 14 12:00:11 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.41299999999999998,"15min":0.41099999999999998},"min":{"1min":0.313,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.56499999999999995,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.36699999999999999,"15min":0.36299999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.53100000000000003}]},{"osd":4,"last update":"Sat Oct 14 12:00:11 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.58599999999999997,"15min":0.58199999999999996},"min":{"1min":0.46600000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.83099999999999996,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.59999999999999998,"15min":0.59399999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.81200000000000006}]},{"osd":5,"last update":"Sat Oct 14 12:00:11 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.622,"15min":0.61799999999999999},"min":{"1min":0.47699999999999998,"5min":0.432,"15min":0.432},"max":{"1min":0.80900000000000005,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.67100000000000004,"15min":0.67000000000000004},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.84399999999999997,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.72799999999999998}]},{"osd":6,"last update":"Sat Oct 14 12:00:11 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.626,"15min":0.628},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.81000000000000005,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.625,"15min":0.622},"min":{"1min":0.49199999999999999,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.85799999999999998,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.85799999999999998}]},{"osd":7,"last update":"Sat Oct 14 12:00:11 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64200000000000002,"15min":0.64100000000000001},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.626,"15min":0.624},"min":{"1min":0.48599999999999999,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.79000000000000004,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.82999999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574877,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27476,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":471040,"data_stored":746908,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 12:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.311,"15min":0.31},"min":{"1min":0.22800000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.48199999999999998,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.32800000000000001,"15min":0.32200000000000001},"min":{"1min":0.253,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"last":0.374}]},{"osd":2,"last update":"Sat Oct 14 12:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.33000000000000002,"15min":0.32300000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.36199999999999999,"15min":0.35899999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.42799999999999999}]},{"osd":3,"last update":"Sat Oct 14 12:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.35299999999999998,"15min":0.34499999999999997},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.433,"5min":0.39200000000000002,"15min":0.38500000000000001},"min":{"1min":0.32000000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.56000000000000005}]},{"osd":4,"last update":"Sat Oct 14 12:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.60399999999999998,"15min":0.58599999999999997},"min":{"1min":0.501,"5min":0.442,"15min":0.442},"max":{"1min":2.3839999999999999,"5min":2.3839999999999999,"15min":2.3839999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.59899999999999998,"15min":0.57699999999999996},"min":{"1min":0.46000000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":2.4100000000000001,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":0.74199999999999999}]},{"osd":5,"last update":"Sat Oct 14 12:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.59899999999999998,"15min":0.58999999999999997},"min":{"1min":0.48399999999999999,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.58999999999999997,"15min":0.58199999999999996},"min":{"1min":0.45400000000000001,"5min":0.44,"15min":0.44},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.78100000000000003}]},{"osd":6,"last update":"Sat Oct 14 12:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.56299999999999994,"15min":0.55900000000000005},"min":{"1min":0.46899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.73299999999999998,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.625,"5min":0.60699999999999998,"15min":0.60399999999999998},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.65200000000000002}]},{"osd":7,"last update":"Sat Oct 14 12:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.621,"15min":0.62},"min":{"1min":0.51700000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.627,"5min":0.61299999999999999,"15min":0.61099999999999999},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.81499999999999995}]}]},{"osd":2,"up_from":13,"seq":55834574877,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27420,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":405504,"data_stored":156540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 12:00:10 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.308,"15min":0.29999999999999999},"min":{"1min":0.24099999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.374,"5min":0.36199999999999999,"15min":0.35999999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.54300000000000004,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.43099999999999999}]},{"osd":1,"last update":"Sat Oct 14 12:00:10 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.371,"15min":0.36599999999999999},"min":{"1min":0.27200000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.38200000000000001,"15min":0.378},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.35899999999999999}]},{"osd":3,"last update":"Sat Oct 14 12:00:10 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.34799999999999998,"15min":0.34000000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.58399999999999996,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.377,"15min":0.36899999999999999},"min":{"1min":0.30199999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.59099999999999997,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.49299999999999999}]},{"osd":4,"last update":"Sat Oct 14 12:00:10 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.58299999999999996,"15min":0.57599999999999996},"min":{"1min":0.437,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81000000000000005,"5min":1,"15min":1},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.60599999999999998,"15min":0.60299999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.54900000000000004}]},{"osd":5,"last update":"Sat Oct 14 12:00:10 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.59799999999999998,"15min":0.58999999999999997},"min":{"1min":0.41399999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88100000000000001,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.626},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.59099999999999997,"15min":0.58199999999999996},"min":{"1min":0.42499999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.88200000000000001,"5min":1.036,"15min":1.036},"last":0.58499999999999996}]},{"osd":6,"last update":"Sat Oct 14 12:00:10 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.629,"15min":0.62},"min":{"1min":0.46999999999999997,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.59799999999999998,"15min":0.58799999999999997},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.60299999999999998}]},{"osd":7,"last update":"Sat Oct 14 12:00:10 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.63200000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.84999999999999998,"5min":0.88,"15min":0.88},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.60599999999999998,"15min":0.59999999999999998},"min":{"1min":0.44600000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.92100000000000004,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.56699999999999995}]}]},{"osd":3,"up_from":13,"seq":55834574878,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27416,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":405504,"data_stored":156540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 12:00:11 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.64800000000000002,"15min":0.70099999999999996},"min":{"1min":0.22800000000000001,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.59499999999999997,"5min":7.226,"15min":7.226},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.38300000000000001,"5min":1.3819999999999999,"15min":1.548},"min":{"1min":0.28499999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.70299999999999996,"5min":21.067,"15min":21.067},"last":0.375}]},{"osd":1,"last update":"Sat Oct 14 12:00:11 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.32800000000000001,"15min":0.32200000000000001},"min":{"1min":0.27000000000000002,"5min":0.247,"15min":0.247},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.70399999999999996,"15min":0.754},"min":{"1min":0.30099999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.72199999999999998,"5min":7.3419999999999996,"15min":7.3419999999999996},"last":0.40300000000000002}]},{"osd":2,"last update":"Sat Oct 14 12:00:11 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.70499999999999996,"15min":0.75600000000000001},"min":{"1min":0.312,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.627,"5min":7.3120000000000003,"15min":7.3120000000000003},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.72599999999999998,"15min":0.77700000000000002},"min":{"1min":0.32800000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.76300000000000001,"5min":7.1870000000000003,"15min":7.1870000000000003},"last":0.44600000000000001}]},{"osd":4,"last update":"Sat Oct 14 12:00:11 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":1.611,"15min":1.7789999999999999},"min":{"1min":0.47599999999999998,"5min":0.436,"15min":0.436},"max":{"1min":0.82799999999999996,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.63900000000000001,"5min":1.444,"15min":1.5780000000000001},"min":{"1min":0.48599999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.83999999999999997,"5min":17.902999999999999,"15min":17.902999999999999},"last":0.48799999999999999}]},{"osd":5,"last update":"Sat Oct 14 12:00:11 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.91500000000000004,"15min":0.96299999999999997},"min":{"1min":0.53800000000000003,"5min":0.37,"15min":0.37},"max":{"1min":0.82499999999999996,"5min":7.2560000000000002,"15min":7.2560000000000002},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.63600000000000001,"5min":1.639,"15min":1.806},"min":{"1min":0.46500000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.85299999999999998,"5min":21.100000000000001,"15min":21.100000000000001},"last":0.59699999999999998}]},{"osd":6,"last update":"Sat Oct 14 12:00:11 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.93000000000000005,"15min":0.97799999999999998},"min":{"1min":0.47399999999999998,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.81399999999999995,"5min":7.1589999999999998,"15min":7.1589999999999998},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.95299999999999996,"15min":1.0049999999999999},"min":{"1min":0.47299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.86799999999999999,"5min":7.2800000000000002,"15min":7.2800000000000002},"last":0.61299999999999999}]},{"osd":7,"last update":"Sat Oct 14 12:00:11 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":1.6379999999999999,"15min":1.806},"min":{"1min":0.48599999999999999,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.751,"5min":21.035,"15min":21.035},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.95199999999999996,"15min":1.0029999999999999},"min":{"1min":0.503,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.78300000000000003,"5min":7.1230000000000002,"15min":7.1230000000000002},"last":0.55400000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574878,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27416,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":405504,"data_stored":156540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 12:00:10 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.77600000000000002,"15min":0.80900000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.74299999999999999,"5min":4.6159999999999997,"15min":4.6159999999999997},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57599999999999996,"15min":0.58099999999999996},"min":{"1min":0.38,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.71999999999999997,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.59699999999999998}]},{"osd":1,"last update":"Sat Oct 14 12:00:10 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.63300000000000001,"15min":0.65500000000000003},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.72799999999999998,"5min":1.637,"15min":1.637},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.71299999999999997,"15min":0.73099999999999998},"min":{"1min":0.45000000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.80000000000000004,"5min":3.6600000000000001,"15min":3.6600000000000001},"last":0.67000000000000004}]},{"osd":2,"last update":"Sat Oct 14 12:00:10 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.83799999999999997,"15min":0.88400000000000001},"min":{"1min":0.40699999999999997,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.75800000000000001,"5min":5.7160000000000002,"15min":5.7160000000000002},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.67700000000000005,"15min":0.69499999999999995},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.76300000000000001,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.621}]},{"osd":3,"last update":"Sat Oct 14 12:00:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.75800000000000001,"15min":0.79200000000000004},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.74199999999999999,"5min":3.7549999999999999,"15min":3.7549999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.72199999999999998,"15min":0.74399999999999999},"min":{"1min":0.42699999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.73899999999999999,"5min":1.744,"15min":1.744},"last":0.65200000000000002}]},{"osd":5,"last update":"Sat Oct 14 12:00:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.57299999999999995,"15min":0.57999999999999996},"min":{"1min":0.372,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.83399999999999996,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.61099999999999999,"15min":0.62},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.79400000000000004,"5min":0.996,"15min":0.996},"last":0.48699999999999999}]},{"osd":6,"last update":"Sat Oct 14 12:00:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.66100000000000003,"15min":0.68100000000000005},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.81499999999999995,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.46200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.93200000000000005,"5min":1.048,"15min":1.048},"last":0.45300000000000001}]},{"osd":7,"last update":"Sat Oct 14 12:00:10 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.65100000000000002,"15min":0.66200000000000003},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.749,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.622,"5min":0.71899999999999997,"15min":0.73499999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.89300000000000002,"5min":1.131,"15min":1.131},"last":0.52100000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574878,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27420,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":405504,"data_stored":156540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 12:00:13 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.48399999999999999,"15min":0.47099999999999997},"min":{"1min":0.45600000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68400000000000005,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.54200000000000004,"15min":0.53800000000000003},"min":{"1min":0.43099999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.72399999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.54400000000000004}]},{"osd":1,"last update":"Sat Oct 14 12:00:13 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.55300000000000005,"15min":0.54800000000000004},"min":{"1min":0.40899999999999997,"5min":0.311,"15min":0.311},"max":{"1min":0.71599999999999997,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.51900000000000002,"15min":0.51300000000000001},"min":{"1min":0.39600000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.74299999999999999,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.53100000000000003}]},{"osd":2,"last update":"Sat Oct 14 12:00:13 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.55800000000000005,"15min":0.55200000000000005},"min":{"1min":0.41399999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.67600000000000005,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.52600000000000002,"15min":0.52000000000000002},"min":{"1min":0.434,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.59699999999999998}]},{"osd":3,"last update":"Sat Oct 14 12:00:13 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.56200000000000006,"15min":0.55500000000000005},"min":{"1min":0.40999999999999998,"5min":0.371,"15min":0.371},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.629,"5min":0.59499999999999997,"15min":0.58899999999999997},"min":{"1min":0.48699999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77600000000000002,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.60899999999999999}]},{"osd":4,"last update":"Sat Oct 14 12:00:13 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51500000000000001,"15min":0.51600000000000001},"min":{"1min":0.33400000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.378},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.57899999999999996,"15min":0.58699999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.41599999999999998}]},{"osd":6,"last update":"Sat Oct 14 12:00:13 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.58699999999999997,"15min":0.58799999999999997},"min":{"1min":0.42699999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.76600000000000001,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.60899999999999999,"15min":0.61099999999999999},"min":{"1min":0.45200000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.89300000000000002,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.503}]},{"osd":7,"last update":"Sat Oct 14 12:00:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.877,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.64000000000000001,"15min":0.64400000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.83899999999999997,"5min":1,"15min":1},"last":0.48299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32622,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T12:00:27.156 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-14T12:00:27.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.274+0000 7fde08a86700 1 Processor -- start 2023-10-14T12:00:27.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.274+0000 7fde08a86700 1 -- start start 2023-10-14T12:00:27.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.275+0000 7fde08a86700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fde0413dbe0 0x7fde0406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:27.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.275+0000 7fde08a86700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde0406ebe0 con 0x7fde040c4540 2023-10-14T12:00:27.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.275+0000 7fde08a86700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde041558c0 con 0x7fde0406ae10 2023-10-14T12:00:27.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.275+0000 7fde08a86700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde0406d330 con 0x7fde0413dbe0 2023-10-14T12:00:27.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.275+0000 7fde0259c700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fde0413dbe0 0x7fde0406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:27.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.275+0000 7fde01d9b700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fde040c4540 0x7fde040c02e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52328/0 (socket says 172.21.15.78:52328) 2023-10-14T12:00:27.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.275+0000 7fde01d9b700 1 -- 172.21.15.78:0/3856111333 learned_addr learned my addr 172.21.15.78:0/3856111333 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T12:00:27.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.275+0000 7fde0259c700 1 -- 172.21.15.78:0/3856111333 >> v1:172.21.15.195:6789/0 conn(0x7fde0406ae10 legacy=0x7fde0415f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T12:00:27.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.275+0000 7fde0259c700 1 -- 172.21.15.78:0/3856111333 >> v1:172.21.15.78:6789/0 conn(0x7fde040c4540 legacy=0x7fde040c02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:27.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.275+0000 7fde0259c700 1 -- 172.21.15.78:0/3856111333 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde04154b00 con 0x7fde0413dbe0 2023-10-14T12:00:27.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.275+0000 7fde0259c700 1 --2- 172.21.15.78:0/3856111333 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde0413dbe0 0x7fde0406a8d0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fddf8009840 tx=0x7fddf8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1cf39cd507d4f6c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:27.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.275+0000 7fde0159a700 1 -- 172.21.15.78:0/3856111333 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf8004230 con 0x7fde0413dbe0 2023-10-14T12:00:27.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.275+0000 7fde0159a700 1 -- 172.21.15.78:0/3856111333 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf8004390 con 0x7fde0413dbe0 2023-10-14T12:00:27.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.275+0000 7fde0159a700 1 -- 172.21.15.78:0/3856111333 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf80036a0 con 0x7fde0413dbe0 2023-10-14T12:00:27.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.276+0000 7fde08a86700 1 -- 172.21.15.78:0/3856111333 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde0413dbe0 msgr2=0x7fde0406a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:27.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.276+0000 7fde08a86700 1 --2- 172.21.15.78:0/3856111333 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde0413dbe0 0x7fde0406a8d0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fddf8009840 tx=0x7fddf8009b50 comp rx=0 tx=0).stop 2023-10-14T12:00:27.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.276+0000 7fde08a86700 1 -- 172.21.15.78:0/3856111333 shutdown_connections 2023-10-14T12:00:27.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.276+0000 7fde08a86700 1 --2- 172.21.15.78:0/3856111333 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde0413dbe0 0x7fde0406a8d0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:27.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.276+0000 7fde08a86700 1 -- 172.21.15.78:0/3856111333 >> 172.21.15.78:0/3856111333 conn(0x7fde040c3a50 msgr2=0x7fde040c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:27.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.276+0000 7fde08a86700 1 -- 172.21.15.78:0/3856111333 shutdown_connections 2023-10-14T12:00:27.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.276+0000 7fde08a86700 1 -- 172.21.15.78:0/3856111333 wait complete. 2023-10-14T12:00:27.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.277+0000 7fde08a86700 1 Processor -- start 2023-10-14T12:00:27.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.277+0000 7fde08a86700 1 -- start start 2023-10-14T12:00:27.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.278+0000 7fde08a86700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fde0406ae10 0x7fde041ed1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:27.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.278+0000 7fde08a86700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fde040c4540 0x7fde04156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:27.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.278+0000 7fde08a86700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde0413dbe0 0x7fde04157430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:27.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.278+0000 7fde08a86700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde04157c30 con 0x7fde0413dbe0 2023-10-14T12:00:27.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.278+0000 7fde08a86700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde04158e10 con 0x7fde0406ae10 2023-10-14T12:00:27.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.278+0000 7fde08a86700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde041ed9e0 con 0x7fde040c4540 2023-10-14T12:00:27.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.278+0000 7fde02d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde0413dbe0 0x7fde04157430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:27.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.278+0000 7fde02d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde0413dbe0 0x7fde04157430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47462/0 (socket says 172.21.15.78:47462) 2023-10-14T12:00:27.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.278+0000 7fde02d9d700 1 -- 172.21.15.78:0/3809699193 learned_addr learned my addr 172.21.15.78:0/3809699193 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:27.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.278+0000 7fde0259c700 1 --2- 172.21.15.78:0/3809699193 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fde0406ae10 0x7fde041ed1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:27.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.278+0000 7fde01d9b700 1 --2- 172.21.15.78:0/3809699193 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fde040c4540 0x7fde04156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:27.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.278+0000 7fde02d9d700 1 -- 172.21.15.78:0/3809699193 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fde040c4540 msgr2=0x7fde04156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:27.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.278+0000 7fde02d9d700 1 --2- 172.21.15.78:0/3809699193 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fde040c4540 0x7fde04156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:27.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.278+0000 7fde02d9d700 1 -- 172.21.15.78:0/3809699193 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fde0406ae10 msgr2=0x7fde041ed1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:27.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.278+0000 7fde02d9d700 1 --2- 172.21.15.78:0/3809699193 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fde0406ae10 0x7fde041ed1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:27.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.278+0000 7fde02d9d700 1 -- 172.21.15.78:0/3809699193 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf80094a0 con 0x7fde0413dbe0 2023-10-14T12:00:27.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.279+0000 7fde02d9d700 1 --2- 172.21.15.78:0/3809699193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde0413dbe0 0x7fde04157430 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fddfc00b9e0 tx=0x7fddfc00bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:27.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.279+0000 7fddf2ffd700 1 -- 172.21.15.78:0/3809699193 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddfc00c9b0 con 0x7fde0413dbe0 2023-10-14T12:00:27.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.279+0000 7fddf2ffd700 1 -- 172.21.15.78:0/3809699193 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddfc00cb10 con 0x7fde0413dbe0 2023-10-14T12:00:27.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.279+0000 7fde08a86700 1 -- 172.21.15.78:0/3809699193 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde041edcc0 con 0x7fde0413dbe0 2023-10-14T12:00:27.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.279+0000 7fddf2ffd700 1 -- 172.21.15.78:0/3809699193 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddfc014440 con 0x7fde0413dbe0 2023-10-14T12:00:27.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.279+0000 7fde08a86700 1 -- 172.21.15.78:0/3809699193 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde04137820 con 0x7fde0413dbe0 2023-10-14T12:00:27.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.280+0000 7fde08a86700 1 -- 172.21.15.78:0/3809699193 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde0404ebb0 con 0x7fde0413dbe0 2023-10-14T12:00:27.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.280+0000 7fddf2ffd700 1 -- 172.21.15.78:0/3809699193 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fddfc006d50 con 0x7fde0413dbe0 2023-10-14T12:00:27.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.281+0000 7fddf2ffd700 1 --2- 172.21.15.78:0/3809699193 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fdde406af00 0x7fdde406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:27.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.281+0000 7fde0259c700 1 --2- 172.21.15.78:0/3809699193 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fdde406af00 0x7fdde406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:27.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.281+0000 7fddf2ffd700 1 -- 172.21.15.78:0/3809699193 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fddfc088d10 con 0x7fde0413dbe0 2023-10-14T12:00:27.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.281+0000 7fde0259c700 1 --2- 172.21.15.78:0/3809699193 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fdde406af00 0x7fdde406d3c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fddf8000c00 tx=0x7fddf801c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:27.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.285+0000 7fddf2ffd700 1 -- 172.21.15.78:0/3809699193 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fddfc0540c0 con 0x7fde0413dbe0 2023-10-14T12:00:27.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.474+0000 7fde08a86700 1 -- 172.21.15.78:0/3809699193 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fde04074b80 con 0x7fde0413dbe0 2023-10-14T12:00:27.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.477+0000 7fddf2ffd700 1 -- 172.21.15.78:0/3809699193 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v15)=0 v15) v1 ==== 123+0+0 (secure 0 0 0) 0x7fddfc0190c0 con 0x7fde0413dbe0 2023-10-14T12:00:27.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.478+0000 7fddf2ffd700 1 -- 172.21.15.78:0/3809699193 <== mon.0 v2:172.21.15.78:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fddfc009820 con 0x7fde0413dbe0 2023-10-14T12:00:27.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.479+0000 7fde08a86700 1 -- 172.21.15.78:0/3809699193 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fdde406af00 msgr2=0x7fdde406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:27.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.479+0000 7fde08a86700 1 --2- 172.21.15.78:0/3809699193 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fdde406af00 0x7fdde406d3c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fddf8000c00 tx=0x7fddf801c040 comp rx=0 tx=0).stop 2023-10-14T12:00:27.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.479+0000 7fde08a86700 1 -- 172.21.15.78:0/3809699193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde0413dbe0 msgr2=0x7fde04157430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:27.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.479+0000 7fde08a86700 1 --2- 172.21.15.78:0/3809699193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde0413dbe0 0x7fde04157430 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fddfc00b9e0 tx=0x7fddfc00bda0 comp rx=0 tx=0).stop 2023-10-14T12:00:27.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.480+0000 7fde08a86700 1 -- 172.21.15.78:0/3809699193 shutdown_connections 2023-10-14T12:00:27.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.480+0000 7fde08a86700 1 --2- 172.21.15.78:0/3809699193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde0413dbe0 0x7fde04157430 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:27.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.480+0000 7fde08a86700 1 --2- 172.21.15.78:0/3809699193 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fdde406af00 0x7fdde406d3c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:27.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.480+0000 7fde08a86700 1 --2- 172.21.15.78:0/3809699193 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fde0406ae10 0x7fde041ed1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:27.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.480+0000 7fde08a86700 1 --2- 172.21.15.78:0/3809699193 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fde040c4540 0x7fde04156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:27.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.480+0000 7fde08a86700 1 -- 172.21.15.78:0/3809699193 >> 172.21.15.78:0/3809699193 conn(0x7fde040c3a50 msgr2=0x7fde040c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:27.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.480+0000 7fde08a86700 1 -- 172.21.15.78:0/3809699193 shutdown_connections 2023-10-14T12:00:27.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.480+0000 7fde08a86700 1 -- 172.21.15.78:0/3809699193 wait complete. 2023-10-14T12:00:27.493 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-14T12:00:27.494 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T12:00:27.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.611+0000 7fde3bd7d700 1 Processor -- start 2023-10-14T12:00:27.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.611+0000 7fde3bd7d700 1 -- start start 2023-10-14T12:00:27.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.612+0000 7fde3bd7d700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fde3406ab40 0x7fde3406f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:27.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.612+0000 7fde3bd7d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fde340c0350 0x7fde3413d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:27.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.612+0000 7fde3bd7d700 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde3406b630 con 0x7fde341554e0 2023-10-14T12:00:27.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.612+0000 7fde3bd7d700 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde340c49f0 con 0x7fde3406ab40 2023-10-14T12:00:27.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.612+0000 7fde3bd7d700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde340726a0 con 0x7fde340c0350 2023-10-14T12:00:27.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.612+0000 7fde39b19700 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fde3406ab40 0x7fde3406f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:27.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.612+0000 7fde39318700 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fde341554e0 0x7fde3406af20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52342/0 (socket says 172.21.15.78:52342) 2023-10-14T12:00:27.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.612+0000 7fde39318700 1 -- 172.21.15.78:0/822717335 learned_addr learned my addr 172.21.15.78:0/822717335 (peer_addr_for_me v1:172.21.15.78:0/0) 2023-10-14T12:00:27.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.612+0000 7fde39b19700 1 -- 172.21.15.78:0/822717335 >> v2:172.21.15.195:3300/0 conn(0x7fde340c0350 msgr2=0x7fde3413d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T12:00:27.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.612+0000 7fde39b19700 1 --2- 172.21.15.78:0/822717335 >> v2:172.21.15.195:3300/0 conn(0x7fde340c0350 0x7fde3413d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:27.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.612+0000 7fde39b19700 1 -- 172.21.15.78:0/822717335 >> v1:172.21.15.78:6789/0 conn(0x7fde341554e0 legacy=0x7fde3406af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:27.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.612+0000 7fde39b19700 1 -- 172.21.15.78:0/822717335 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde34072980 con 0x7fde3406ab40 2023-10-14T12:00:27.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.612+0000 7fde38b17700 1 -- 172.21.15.78:0/822717335 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 375537843 0 0) 0x7fde3406b630 con 0x7fde341554e0 2023-10-14T12:00:27.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.613+0000 7fde39b19700 1 --2- 172.21.15.78:0/822717335 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde3406ab40 0x7fde3406f1e0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fde340c1860 tx=0x7fde24009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92034db4e5ba76a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:27.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.613+0000 7fde38b17700 1 -- 172.21.15.78:0/822717335 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde24011830 con 0x7fde3406ab40 2023-10-14T12:00:27.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.613+0000 7fde38b17700 1 -- 172.21.15.78:0/822717335 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fde24011990 con 0x7fde3406ab40 2023-10-14T12:00:27.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.613+0000 7fde38b17700 1 -- 172.21.15.78:0/822717335 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde24011d40 con 0x7fde3406ab40 2023-10-14T12:00:27.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.613+0000 7fde3bd7d700 1 -- 172.21.15.78:0/822717335 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde3406ab40 msgr2=0x7fde3406f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:27.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.613+0000 7fde3bd7d700 1 --2- 172.21.15.78:0/822717335 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde3406ab40 0x7fde3406f1e0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fde340c1860 tx=0x7fde24009b20 comp rx=0 tx=0).stop 2023-10-14T12:00:27.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.613+0000 7fde3bd7d700 1 -- 172.21.15.78:0/822717335 shutdown_connections 2023-10-14T12:00:27.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.613+0000 7fde3bd7d700 1 --2- 172.21.15.78:0/822717335 >> v2:172.21.15.195:3300/0 conn(0x7fde340c0350 0x7fde3413d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:27.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.613+0000 7fde3bd7d700 1 --2- 172.21.15.78:0/822717335 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde3406ab40 0x7fde3406f1e0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:27.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.613+0000 7fde3bd7d700 1 -- 172.21.15.78:0/822717335 >> 172.21.15.78:0/822717335 conn(0x7fde340c3ac0 msgr2=0x7fde340c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:27.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.613+0000 7fde3bd7d700 1 -- 172.21.15.78:0/822717335 shutdown_connections 2023-10-14T12:00:27.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.613+0000 7fde3bd7d700 1 -- 172.21.15.78:0/822717335 wait complete. 2023-10-14T12:00:27.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.614+0000 7fde3bd7d700 1 Processor -- start 2023-10-14T12:00:27.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.614+0000 7fde3bd7d700 1 -- start start 2023-10-14T12:00:27.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.615+0000 7fde3bd7d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde3406ab40 0x7fde34151f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:27.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.615+0000 7fde3bd7d700 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fde340c0350 0x7fde34158680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:27.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.615+0000 7fde3bd7d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fde341554e0 0x7fde340aba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:27.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.615+0000 7fde3bd7d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde3414f5e0 con 0x7fde3406ab40 2023-10-14T12:00:27.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.615+0000 7fde3bd7d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde34158bc0 con 0x7fde341554e0 2023-10-14T12:00:27.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.615+0000 7fde3bd7d700 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde340abfc0 con 0x7fde340c0350 2023-10-14T12:00:27.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.615+0000 7fde39b19700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde3406ab40 0x7fde34151f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:27.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.615+0000 7fde39b19700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde3406ab40 0x7fde34151f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47472/0 (socket says 172.21.15.78:47472) 2023-10-14T12:00:27.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.615+0000 7fde39b19700 1 -- 172.21.15.78:0/3123476654 learned_addr learned my addr 172.21.15.78:0/3123476654 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-14T12:00:27.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.615+0000 7fde3a31a700 1 --2- 172.21.15.78:0/3123476654 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fde341554e0 0x7fde340aba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:27.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.615+0000 7fde39318700 1 --2- 172.21.15.78:0/3123476654 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fde340c0350 0x7fde34158680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:27.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.615+0000 7fde39b19700 1 -- 172.21.15.78:0/3123476654 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fde340c0350 msgr2=0x7fde34158680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:27.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.615+0000 7fde39b19700 1 --2- 172.21.15.78:0/3123476654 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fde340c0350 0x7fde34158680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:27.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.615+0000 7fde39b19700 1 -- 172.21.15.78:0/3123476654 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fde341554e0 msgr2=0x7fde340aba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:27.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.615+0000 7fde39b19700 1 --2- 172.21.15.78:0/3123476654 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fde341554e0 0x7fde340aba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:27.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.615+0000 7fde39b19700 1 -- 172.21.15.78:0/3123476654 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde240094a0 con 0x7fde3406ab40 2023-10-14T12:00:27.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.616+0000 7fde39b19700 1 --2- 172.21.15.78:0/3123476654 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde3406ab40 0x7fde34151f80 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fde340ad290 tx=0x7fde24004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:27.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.616+0000 7fde2a7fc700 1 -- 172.21.15.78:0/3123476654 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde240038a0 con 0x7fde3406ab40 2023-10-14T12:00:27.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.616+0000 7fde2a7fc700 1 -- 172.21.15.78:0/3123476654 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fde2401f070 con 0x7fde3406ab40 2023-10-14T12:00:27.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.616+0000 7fde3bd7d700 1 -- 172.21.15.78:0/3123476654 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde3414efb0 con 0x7fde3406ab40 2023-10-14T12:00:27.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.616+0000 7fde2a7fc700 1 -- 172.21.15.78:0/3123476654 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde24003c40 con 0x7fde3406ab40 2023-10-14T12:00:27.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.616+0000 7fde3bd7d700 1 -- 172.21.15.78:0/3123476654 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde341edc40 con 0x7fde3406ab40 2023-10-14T12:00:27.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.617+0000 7fde3bd7d700 1 -- 172.21.15.78:0/3123476654 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde3414f720 con 0x7fde3406ab40 2023-10-14T12:00:27.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.617+0000 7fde2a7fc700 1 -- 172.21.15.78:0/3123476654 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fde2401a430 con 0x7fde3406ab40 2023-10-14T12:00:27.633 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.617+0000 7fde2a7fc700 1 --2- 172.21.15.78:0/3123476654 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fde1806af00 0x7fde1806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T12:00:27.633 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.618+0000 7fde2a7fc700 1 -- 172.21.15.78:0/3123476654 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fde24021070 con 0x7fde3406ab40 2023-10-14T12:00:27.633 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.618+0000 7fde39318700 1 --2- 172.21.15.78:0/3123476654 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fde1806af00 0x7fde1806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T12:00:27.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.618+0000 7fde39318700 1 --2- 172.21.15.78:0/3123476654 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fde1806af00 0x7fde1806d3c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fde3414e8a0 tx=0x7fde30007c90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T12:00:27.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.622+0000 7fde2a7fc700 1 -- 172.21.15.78:0/3123476654 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fde3414f720 con 0x7fde3406ab40 2023-10-14T12:00:27.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.854+0000 7fde3bd7d700 1 -- 172.21.15.78:0/3123476654 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde3414f720 con 0x7fde3406ab40 2023-10-14T12:00:27.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.855+0000 7fde2a7fc700 1 -- 172.21.15.78:0/3123476654 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+3702 (secure 0 0 0) 0x7fde2402b090 con 0x7fde3406ab40 2023-10-14T12:00:27.857 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-14T12:00:27.858 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":19,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4658,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6834/1492486786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1492486786},{"type":"v1","addr":"172.21.15.195:6834","nonce":1492486786}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4928,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/305816559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6833","nonce":305816559},{"type":"v1","addr":"172.21.15.195:6835","nonce":305816559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19},{"gid":5016,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/2991384297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2991384297},{"type":"v1","addr":"172.21.15.78:6836","nonce":2991384297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5019,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3246640037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3246640037},{"type":"v1","addr":"172.21.15.78:6837","nonce":3246640037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[]} 2023-10-14T12:00:27.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.856+0000 7fde3bd7d700 1 -- 172.21.15.78:0/3123476654 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fde1806af00 msgr2=0x7fde1806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:27.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.857+0000 7fde3bd7d700 1 --2- 172.21.15.78:0/3123476654 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fde1806af00 0x7fde1806d3c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fde3414e8a0 tx=0x7fde30007c90 comp rx=0 tx=0).stop 2023-10-14T12:00:27.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.857+0000 7fde3bd7d700 1 -- 172.21.15.78:0/3123476654 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde3406ab40 msgr2=0x7fde34151f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T12:00:27.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.857+0000 7fde3bd7d700 1 --2- 172.21.15.78:0/3123476654 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde3406ab40 0x7fde34151f80 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fde340ad290 tx=0x7fde24004750 comp rx=0 tx=0).stop 2023-10-14T12:00:27.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.857+0000 7fde3bd7d700 1 -- 172.21.15.78:0/3123476654 shutdown_connections 2023-10-14T12:00:27.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.857+0000 7fde3bd7d700 1 --2- 172.21.15.78:0/3123476654 >> [v2:172.21.15.78:6832/3171890074,v1:172.21.15.78:6833/3171890074] conn(0x7fde1806af00 0x7fde1806d3c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:27.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.857+0000 7fde3bd7d700 1 --2- 172.21.15.78:0/3123476654 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fde341554e0 0x7fde340aba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:27.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.857+0000 7fde3bd7d700 1 --2- 172.21.15.78:0/3123476654 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde3406ab40 0x7fde34151f80 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:27.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.857+0000 7fde3bd7d700 1 --2- 172.21.15.78:0/3123476654 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fde340c0350 0x7fde34158680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T12:00:27.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.857+0000 7fde3bd7d700 1 -- 172.21.15.78:0/3123476654 >> 172.21.15.78:0/3123476654 conn(0x7fde340c3ac0 msgr2=0x7fde340c18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T12:00:27.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.857+0000 7fde3bd7d700 1 -- 172.21.15.78:0/3123476654 shutdown_connections 2023-10-14T12:00:27.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-14T12:00:27.857+0000 7fde3bd7d700 1 -- 172.21.15.78:0/3123476654 wait complete. 2023-10-14T12:00:27.862 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 19 2023-10-14T12:00:27.870 DEBUG:tasks.cephfs.filesystem:already dead... 2023-10-14T12:00:27.871 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-14T12:00:27.872 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-14T12:00:27.872 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T12:00:27.880 INFO:tasks.ceph.mds.a:Stopped 2023-10-14T12:00:27.880 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-14T12:00:27.881 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T12:00:27.928 INFO:tasks.ceph.mds.c:Stopped 2023-10-14T12:00:27.929 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-14T12:00:27.929 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T12:00:27.940 INFO:tasks.ceph.mds.b:Stopped 2023-10-14T12:00:27.941 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-14T12:00:27.941 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T12:00:27.992 INFO:tasks.ceph.mds.d:Stopped 2023-10-14T12:00:27.993 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-14T12:00:27.993 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-14T12:00:27.993 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T12:00:28.021 INFO:tasks.ceph.osd.0:Stopped 2023-10-14T12:00:28.022 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-14T12:00:28.022 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T12:00:28.092 INFO:tasks.ceph.osd.1:Stopped 2023-10-14T12:00:28.092 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-14T12:00:28.093 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T12:00:28.158 INFO:tasks.ceph.osd.2:Stopped 2023-10-14T12:00:28.159 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-10-14T12:00:28.159 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T12:00:28.233 INFO:tasks.ceph.osd.3:Stopped 2023-10-14T12:00:28.233 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-10-14T12:00:28.234 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T12:00:28.278 INFO:tasks.ceph.osd.4:Stopped 2023-10-14T12:00:28.279 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-10-14T12:00:28.279 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T12:00:28.321 INFO:tasks.ceph.osd.5:Stopped 2023-10-14T12:00:28.322 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-10-14T12:00:28.322 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T12:00:28.361 INFO:tasks.ceph.osd.6:Stopped 2023-10-14T12:00:28.362 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-10-14T12:00:28.362 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T12:00:28.437 INFO:tasks.ceph.osd.7:Stopped 2023-10-14T12:00:28.438 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-14T12:00:28.438 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-10-14T12:00:28.438 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T12:00:28.462 INFO:tasks.ceph.mgr.y:Stopped 2023-10-14T12:00:28.462 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-14T12:00:28.463 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T12:00:28.504 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2023-10-14T12:00:28.505 INFO:tasks.ceph.mgr.x:Stopped 2023-10-14T12:00:28.505 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-14T12:00:28.506 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-14T12:00:28.506 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T12:00:28.516 INFO:tasks.ceph.mon.a:Stopped 2023-10-14T12:00:28.517 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-10-14T12:00:28.517 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T12:00:28.556 INFO:tasks.ceph.mon.b:Stopped 2023-10-14T12:00:28.556 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-10-14T12:00:28.557 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T12:00:28.607 INFO:tasks.ceph.mon.c:Stopped 2023-10-14T12:00:28.608 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-14T12:00:28.609 DEBUG:teuthology.orchestra.run.smithi078:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'Replacing daemon mds' | egrep -v '\(MDS_INSUFFICIENT_STANDBY\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(FS_DEGRADED\)' | head -n 1 2023-10-14T12:00:28.641 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi078.front.sepia.ceph.com 2023-10-14T12:00:28.642 DEBUG:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-14T12:00:28.912 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi078.front.sepia.ceph.com 2023-10-14T12:00:28.913 DEBUG:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-14T12:00:29.020 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi078.front.sepia.ceph.com 2023-10-14T12:00:29.021 DEBUG:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-14T12:00:29.102 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi078.front.sepia.ceph.com 2023-10-14T12:00:29.103 DEBUG:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-10-14T12:00:29.177 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi195.front.sepia.ceph.com 2023-10-14T12:00:29.178 DEBUG:teuthology.orchestra.run.smithi195:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-10-14T12:00:29.476 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi195.front.sepia.ceph.com 2023-10-14T12:00:29.476 DEBUG:teuthology.orchestra.run.smithi195:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-10-14T12:00:29.557 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi195.front.sepia.ceph.com 2023-10-14T12:00:29.558 DEBUG:teuthology.orchestra.run.smithi195:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-10-14T12:00:29.649 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi195.front.sepia.ceph.com 2023-10-14T12:00:29.649 DEBUG:teuthology.orchestra.run.smithi195:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-10-14T12:00:29.750 INFO:tasks.ceph:Archiving mon data... 2023-10-14T12:00:29.752 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rishabh-2023-10-13_18:04:14-fs-rishabh-2023oct13-testing-default-smithi/7425292/data/mon.a.tgz 2023-10-14T12:00:29.753 DEBUG:teuthology.orchestra.run.smithi078:> mktemp 2023-10-14T12:00:29.771 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/tmp.F5GivHtAeX 2023-10-14T12:00:29.771 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.F5GivHtAeX 2023-10-14T12:00:29.893 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0666 /tmp/tmp.F5GivHtAeX 2023-10-14T12:00:29.980 DEBUG:teuthology.orchestra.remote:smithi078:/tmp/tmp.F5GivHtAeX is 282KB 2023-10-14T12:00:30.048 DEBUG:teuthology.orchestra.run.smithi078:> rm -fr /tmp/tmp.F5GivHtAeX 2023-10-14T12:00:30.065 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/rishabh-2023-10-13_18:04:14-fs-rishabh-2023oct13-testing-default-smithi/7425292/data/mon.b.tgz 2023-10-14T12:00:30.065 DEBUG:teuthology.orchestra.run.smithi195:> mktemp 2023-10-14T12:00:30.082 INFO:teuthology.orchestra.run.smithi195.stdout:/tmp/tmp.Z6K2PHx9ds 2023-10-14T12:00:30.083 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.Z6K2PHx9ds 2023-10-14T12:00:30.224 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0666 /tmp/tmp.Z6K2PHx9ds 2023-10-14T12:00:30.311 DEBUG:teuthology.orchestra.remote:smithi195:/tmp/tmp.Z6K2PHx9ds is 381KB 2023-10-14T12:00:30.407 DEBUG:teuthology.orchestra.run.smithi195:> rm -fr /tmp/tmp.Z6K2PHx9ds 2023-10-14T12:00:30.424 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/rishabh-2023-10-13_18:04:14-fs-rishabh-2023oct13-testing-default-smithi/7425292/data/mon.c.tgz 2023-10-14T12:00:30.424 DEBUG:teuthology.orchestra.run.smithi195:> mktemp 2023-10-14T12:00:30.480 INFO:teuthology.orchestra.run.smithi195.stdout:/tmp/tmp.Xliat2TEQR 2023-10-14T12:00:30.481 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.Xliat2TEQR 2023-10-14T12:00:30.612 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0666 /tmp/tmp.Xliat2TEQR 2023-10-14T12:00:30.697 DEBUG:teuthology.orchestra.remote:smithi195:/tmp/tmp.Xliat2TEQR is 381KB 2023-10-14T12:00:30.792 DEBUG:teuthology.orchestra.run.smithi195:> rm -fr /tmp/tmp.Xliat2TEQR 2023-10-14T12:00:30.809 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-14T12:00:30.810 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-14T12:00:30.812 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-14T12:00:30.875 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-14T12:00:30.875 DEBUG:teuthology.orchestra.run.smithi078:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-14T12:00:30.883 DEBUG:teuthology.orchestra.run.smithi195:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-14T12:00:30.930 INFO:teuthology.orchestra.run.smithi078.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-14T12:00:30.949 INFO:teuthology.orchestra.run.smithi195.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-14T12:00:30.955 INFO:tasks.ceph:Archiving crash dumps... 2023-10-14T12:00:30.957 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2023-10-13_18:04:14-fs-rishabh-2023oct13-testing-default-smithi/7425292/remote/smithi078/crash 2023-10-14T12:00:30.958 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-14T12:00:30.986 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2023-10-13_18:04:14-fs-rishabh-2023oct13-testing-default-smithi/7425292/remote/smithi195/crash 2023-10-14T12:00:30.987 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-14T12:00:31.021 INFO:tasks.ceph:Shutting down logrotate 2023-10-14T12:00:31.022 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T12:00:31.031 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T12:00:31.121 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-14T12:00:31.149 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-14T12:00:31.175 INFO:tasks.ceph:Compressing logs... 2023-10-14T12:00:31.176 DEBUG:teuthology.orchestra.run.smithi078:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-14T12:00:31.192 DEBUG:teuthology.orchestra.run.smithi195:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-14T12:00:31.217 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.111756.log 2023-10-14T12:00:31.217 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108869.log 2023-10-14T12:00:31.233 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.0.111756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109497.log 2023-10-14T12:00:31.234 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.111756.log.gz 2023-10-14T12:00:31.234 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114188.log 2023-10-14T12:00:31.234 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.108869.log: /var/log/ceph/ceph-client.admin.109497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110236.log 2023-10-14T12:00:31.235 INFO:teuthology.orchestra.run.smithi078.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.108869.log.gz 2023-10-14T12:00:31.235 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.114188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110225.log 2023-10-14T12:00:31.235 INFO:teuthology.orchestra.run.smithi078.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.109497.log.gz 2023-10-14T12:00:31.235 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.110236.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.114188.log.gz 2023-10-14T12:00:31.236 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110878.log 2023-10-14T12:00:31.236 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.110236.log.gz 2023-10-14T12:00:31.236 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111108.log 2023-10-14T12:00:31.237 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.110225.log: /var/log/ceph/ceph-client.admin.110878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114124.log 2023-10-14T12:00:31.237 INFO:teuthology.orchestra.run.smithi078.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.110225.log.gz 2023-10-14T12:00:31.237 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.111108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113035.log 2023-10-14T12:00:31.237 INFO:teuthology.orchestra.run.smithi078.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.110878.log.gz 2023-10-14T12:00:31.238 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.114124.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.111108.log.gz 2023-10-14T12:00:31.238 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113802.log 2023-10-14T12:00:31.238 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.113035.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.114124.log.gz 2023-10-14T12:00:31.239 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110602.log 2023-10-14T12:00:31.239 INFO:teuthology.orchestra.run.smithi078.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.113035.log.gz 2023-10-14T12:00:31.239 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.113802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116263.log 2023-10-14T12:00:31.239 INFO:teuthology.orchestra.run.smithi078.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.113802.log.gz 2023-10-14T12:00:31.240 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.110602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112574.log 2023-10-14T12:00:31.240 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.116263.log: 88.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.110602.log.gz 2023-10-14T12:00:31.240 INFO:teuthology.orchestra.run.smithi078.stderr:-5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-14T12:00:31.241 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.112574.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.116263.log.gz 2023-10-14T12:00:31.241 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118265.log 2023-10-14T12:00:31.241 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.112574.log.gz 2023-10-14T12:00:31.241 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118035.log 2023-10-14T12:00:31.242 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.118265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116147.log 2023-10-14T12:00:31.242 INFO:teuthology.orchestra.run.smithi078.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.118265.log.gz 2023-10-14T12:00:31.242 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.118035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118541.log 2023-10-14T12:00:31.243 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.116147.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.118035.log.gz 2023-10-14T12:00:31.243 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114078.log 2023-10-14T12:00:31.243 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.118541.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.116147.log.gz 2023-10-14T12:00:31.243 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110648.log 2023-10-14T12:00:31.244 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.118541.log.gz 2023-10-14T12:00:31.244 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.114078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113162.log 2023-10-14T12:00:31.244 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.114078.log.gz 2023-10-14T12:00:31.245 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.110648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108195.log 2023-10-14T12:00:31.245 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.113162.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.110648.log.gz 2023-10-14T12:00:31.245 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112666.log 2023-10-14T12:00:31.245 INFO:teuthology.orchestra.run.smithi078.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.113162.log.gz 2023-10-14T12:00:31.246 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.108195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117046.log 2023-10-14T12:00:31.246 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.112666.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.108195.log.gz 2023-10-14T12:00:31.246 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115448.log 2023-10-14T12:00:31.247 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.117046.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.112666.log.gz 2023-10-14T12:00:31.247 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-14T12:00:31.247 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.115448.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.117046.log.gz 2023-10-14T12:00:31.247 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113434.log 2023-10-14T12:00:31.248 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.115448.log.gz 2023-10-14T12:00:31.248 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118755.log 2023-10-14T12:00:31.248 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.113434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113848.log 2023-10-14T12:00:31.249 INFO:teuthology.orchestra.run.smithi078.stderr: 85.0% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-14T12:00:31.249 INFO:teuthology.orchestra.run.smithi078.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.113434.log.gz 2023-10-14T12:00:31.249 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.118755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116816.log 2023-10-14T12:00:31.249 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117897.log 2023-10-14T12:00:31.250 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.113848.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.118755.log.gz 2023-10-14T12:00:31.250 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.116816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115402.log 2023-10-14T12:00:31.250 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.113848.log.gz 2023-10-14T12:00:31.251 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.117897.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.116816.log.gz 2023-10-14T12:00:31.251 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118357.log 2023-10-14T12:00:31.251 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.115402.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.117897.log.gz 2023-10-14T12:00:31.251 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116193.log 2023-10-14T12:00:31.252 INFO:teuthology.orchestra.run.smithi078.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.115402.log.gz 2023-10-14T12:00:31.252 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.118357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113618.log 2023-10-14T12:00:31.252 INFO:teuthology.orchestra.run.smithi078.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.118357.log.gz 2023-10-14T12:00:31.253 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.116193.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-14T12:00:31.253 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.113618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.111748.log 2023-10-14T12:00:31.253 INFO:teuthology.orchestra.run.smithi078.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.113618.log.gz 2023-10-14T12:00:31.253 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116056.log 2023-10-14T12:00:31.254 INFO:teuthology.orchestra.run.smithi078.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.116193.log.gz 2023-10-14T12:00:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114032.log 2023-10-14T12:00:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.0.111748.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.111748.log.gz 2023-10-14T12:00:31.255 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.116056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115310.log 2023-10-14T12:00:31.255 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.114032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109855.log 2023-10-14T12:00:31.255 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.114032.log.gz 2023-10-14T12:00:31.255 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.115310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109009.log 2023-10-14T12:00:31.256 INFO:teuthology.orchestra.run.smithi078.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.116056.log.gz 2023-10-14T12:00:31.256 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.115310.log.gz 2023-10-14T12:00:31.256 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110740.log /var/log/ceph/ceph-client.admin.109855.log: 2023-10-14T12:00:31.256 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.109009.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2023-10-14T12:00:31.257 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.109855.log.gz 2023-10-14T12:00:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.110740.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.109009.log.gz 2023-10-14T12:00:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113300.log 2023-10-14T12:00:31.258 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-mds.a.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.110740.log.gz 2023-10-14T12:00:31.258 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114754.log 2023-10-14T12:00:31.258 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109055.log 2023-10-14T12:00:31.258 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.113300.log: /var/log/ceph/ceph-client.admin.114754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117205.log 2023-10-14T12:00:31.259 INFO:teuthology.orchestra.run.smithi078.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.113300.log.gz 2023-10-14T12:00:31.259 INFO:teuthology.orchestra.run.smithi078.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.114754.log.gz 2023-10-14T12:00:31.259 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.109055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113254.log 2023-10-14T12:00:31.260 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117000.log 2023-10-14T12:00:31.260 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.109055.log.gz 2023-10-14T12:00:31.260 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.117205.log: /var/log/ceph/ceph-client.admin.113254.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.117205.log.gz 2023-10-14T12:00:31.261 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.117000.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.113254.log.gz 2023-10-14T12:00:31.261 INFO:teuthology.orchestra.run.smithi078.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.117000.log.gz 2023-10-14T12:00:31.261 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109863.log 2023-10-14T12:00:31.261 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111016.log 2023-10-14T12:00:31.262 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.109863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112528.log 2023-10-14T12:00:31.262 INFO:teuthology.orchestra.run.smithi078.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.109863.log.gz 2023-10-14T12:00:31.262 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.111016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114392.log 2023-10-14T12:00:31.263 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.112528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104046.log 2023-10-14T12:00:31.263 INFO:teuthology.orchestra.run.smithi078.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.111016.log.gz 2023-10-14T12:00:31.263 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.114392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112436.log 2023-10-14T12:00:31.263 INFO:teuthology.orchestra.run.smithi078.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.112528.log.gz 2023-10-14T12:00:31.264 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.104046.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.114392.log.gz 2023-10-14T12:00:31.264 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.112436.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.104046.log.gz 2023-10-14T12:00:31.264 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.112436.log.gz 2023-10-14T12:00:31.265 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-10-14T12:00:31.274 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103881.log 2023-10-14T12:00:31.275 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-10-14T12:00:31.275 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103973.log 2023-10-14T12:00:31.275 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.103881.log: /var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-14T12:00:31.275 INFO:teuthology.orchestra.run.smithi195.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.103881.log.gz 2023-10-14T12:00:31.276 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.103973.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-10-14T12:00:31.276 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.103973.log.gz 2023-10-14T12:00:31.276 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104018.log 2023-10-14T12:00:31.277 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.108581.log 2023-10-14T12:00:31.277 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.104018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104161.log 2023-10-14T12:00:31.277 INFO:teuthology.orchestra.run.smithi195.stderr: 91.6% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-14T12:00:31.277 INFO:teuthology.orchestra.run.smithi195.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.104018.log.gz 2023-10-14T12:00:31.278 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.1.108581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104063.log 2023-10-14T12:00:31.278 INFO:teuthology.orchestra.run.smithi195.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.1.108581.log.gz 2023-10-14T12:00:31.278 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2023-10-14T12:00:31.279 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.104161.log: /var/log/ceph/ceph-client.admin.104063.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-14T12:00:31.279 INFO:teuthology.orchestra.run.smithi195.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.104161.log.gz 2023-10-14T12:00:31.279 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-mds.d.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.104063.log.gz 2023-10-14T12:00:31.279 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103834.log 2023-10-14T12:00:31.280 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-10-14T12:00:31.280 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.103834.log: 88.1%/var/log/ceph/ceph-osd.4.log: -- replaced with /var/log/ceph/ceph-client.admin.103834.log.gz 2023-10-14T12:00:31.280 INFO:teuthology.orchestra.run.smithi195.stderr: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-14T12:00:31.280 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-14T12:00:31.281 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104108.log 2023-10-14T12:00:31.281 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-client.admin.104108.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.104108.log.gz 2023-10-14T12:00:31.281 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-10-14T12:00:31.282 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-10-14T12:00:31.282 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-osd.7.log: 92.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-14T12:00:31.282 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.108573.log 2023-10-14T12:00:31.282 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.1.108573.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.108573.log.gz 2023-10-14T12:00:31.283 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-10-14T12:00:31.283 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-mon.b.log: 91.2% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2023-10-14T12:00:31.283 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2023-10-14T12:00:31.284 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103926.log 2023-10-14T12:00:31.284 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.103926.log: 92.0% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2023-10-14T12:00:31.284 INFO:teuthology.orchestra.run.smithi195.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.103926.log.gz 2023-10-14T12:00:31.313 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115794.log 2023-10-14T12:00:31.314 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108478.log 2023-10-14T12:00:31.314 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.115794.log: /var/log/ceph/ceph-client.admin.108478.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.115794.log.gz 2023-10-14T12:00:31.315 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.108478.log.gz 2023-10-14T12:00:31.315 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110832.log 2023-10-14T12:00:31.315 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118403.log 2023-10-14T12:00:31.315 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.110832.log: /var/log/ceph/ceph-client.admin.118403.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.110832.log.gz 2023-10-14T12:00:31.316 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.118403.log.gz 2023-10-14T12:00:31.316 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110694.log 2023-10-14T12:00:31.316 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116425.log 2023-10-14T12:00:31.317 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.110694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112206.log 2023-10-14T12:00:31.317 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.116425.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.110694.log.gz 2023-10-14T12:00:31.317 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111266.log 2023-10-14T12:00:31.317 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.112206.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.116425.log.gz 2023-10-14T12:00:31.318 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.111266.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.112206.log.gz 2023-10-14T12:00:31.318 INFO:teuthology.orchestra.run.smithi078.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.111266.log.gz 2023-10-14T12:00:31.318 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108531.log 2023-10-14T12:00:31.319 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117943.log 2023-10-14T12:00:31.319 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.108531.log: /var/log/ceph/ceph-client.admin.117943.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.108531.log.gz 2023-10-14T12:00:31.319 INFO:teuthology.orchestra.run.smithi078.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.117943.log.gz 2023-10-14T12:00:31.319 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116379.log 2023-10-14T12:00:31.320 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110235.log 2023-10-14T12:00:31.320 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.116379.log: /var/log/ceph/ceph-client.admin.110235.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.116379.log.gz 2023-10-14T12:00:31.320 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.110235.log.gz 2023-10-14T12:00:31.321 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115080.log 2023-10-14T12:00:31.321 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117712.log 2023-10-14T12:00:31.321 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.115080.log: /var/log/ceph/ceph-client.admin.117712.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.115080.log.gz 2023-10-14T12:00:31.321 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.117712.log.gz 2023-10-14T12:00:31.322 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115172.log 2023-10-14T12:00:31.322 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109293.log 2023-10-14T12:00:31.322 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.115172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112897.log 2023-10-14T12:00:31.322 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.109293.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.115172.log.gz 2023-10-14T12:00:31.323 INFO:teuthology.orchestra.run.smithi078.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.109293.log.gz 2023-10-14T12:00:31.323 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.112897.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.112897.log.gz 2023-10-14T12:00:31.323 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109864.log 2023-10-14T12:00:31.324 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108721.log 2023-10-14T12:00:31.324 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.109864.log: /var/log/ceph/ceph-client.admin.108721.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.109864.log.gz 2023-10-14T12:00:31.324 INFO:teuthology.orchestra.run.smithi078.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.108721.log.gz 2023-10-14T12:00:31.324 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118311.log 2023-10-14T12:00:31.325 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109491.log 2023-10-14T12:00:31.325 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.118311.log: /var/log/ceph/ceph-client.admin.109491.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.118311.log.gz 2023-10-14T12:00:31.325 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.109491.log.gz 2023-10-14T12:00:31.325 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110230.log 2023-10-14T12:00:31.326 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118449.log 2023-10-14T12:00:31.326 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.110230.log: /var/log/ceph/ceph-client.admin.118449.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.110230.log.gz 2023-10-14T12:00:31.326 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.118449.log.gz 2023-10-14T12:00:31.327 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116770.log 2023-10-14T12:00:31.327 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114938.log 2023-10-14T12:00:31.327 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.116770.log: /var/log/ceph/ceph-client.admin.114938.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.116770.log.gz 2023-10-14T12:00:31.327 INFO:teuthology.orchestra.run.smithi078.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.114938.log.gz 2023-10-14T12:00:31.328 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118587.log 2023-10-14T12:00:31.328 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117092.log 2023-10-14T12:00:31.328 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.118587.log: /var/log/ceph/ceph-client.admin.117092.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.118587.log.gz 2023-10-14T12:00:31.328 INFO:teuthology.orchestra.run.smithi078.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.117092.log.gz 2023-10-14T12:00:31.329 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109494.log 2023-10-14T12:00:31.329 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112620.log 2023-10-14T12:00:31.329 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.109494.log: /var/log/ceph/ceph-client.admin.112620.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.109494.log.gz 2023-10-14T12:00:31.330 INFO:teuthology.orchestra.run.smithi078.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.112620.log.gz 2023-10-14T12:00:31.330 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113986.log 2023-10-14T12:00:31.330 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110464.log 2023-10-14T12:00:31.330 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.113986.log: 87.7%/var/log/ceph/ceph-client.admin.110464.log: -- replaced with /var/log/ceph/ceph-client.admin.113986.log.gz 2023-10-14T12:00:31.331 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.110464.log.gz 2023-10-14T12:00:31.331 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2023-10-14T12:00:31.331 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108823.log 2023-10-14T12:00:31.332 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-mds.c.log: /var/log/ceph/ceph-client.admin.108823.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.108823.log.gz 2023-10-14T12:00:31.332 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113710.log 2023-10-14T12:00:31.332 INFO:teuthology.orchestra.run.smithi078.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2023-10-14T12:00:31.332 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115034.log 2023-10-14T12:00:31.333 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.113710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117251.log 2023-10-14T12:00:31.333 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.115034.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.113710.log.gz 2023-10-14T12:00:31.333 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.115034.log.gz 2023-10-14T12:00:31.334 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.117251.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.117251.log.gz 2023-10-14T12:00:31.334 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110556.log 2023-10-14T12:00:31.334 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109858.log 2023-10-14T12:00:31.334 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.110556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112160.log 2023-10-14T12:00:31.335 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.109858.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.110556.log.gz 2023-10-14T12:00:31.335 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.112160.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.109858.log.gz 2023-10-14T12:00:31.335 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.112160.log.gz 2023-10-14T12:00:31.336 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117481.log 2023-10-14T12:00:31.336 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114986.log 2023-10-14T12:00:31.336 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.117481.log: /var/log/ceph/ceph-client.admin.114986.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.117481.log.gz 2023-10-14T12:00:31.336 INFO:teuthology.orchestra.run.smithi078.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.114986.log.gz 2023-10-14T12:00:31.337 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117138.log 2023-10-14T12:00:31.337 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108581.log 2023-10-14T12:00:31.337 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.117138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115126.log 2023-10-14T12:00:31.338 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.108581.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.117138.log.gz 2023-10-14T12:00:31.338 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.115126.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.108581.log.gz 2023-10-14T12:00:31.338 INFO:teuthology.orchestra.run.smithi078.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.115126.log.gz 2023-10-14T12:00:31.338 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109493.log 2023-10-14T12:00:31.339 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112712.log 2023-10-14T12:00:31.339 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.109493.log: /var/log/ceph/ceph-client.admin.112712.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.109493.log.gz 2023-10-14T12:00:31.339 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.112712.log.gz 2023-10-14T12:00:31.340 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116954.log 2023-10-14T12:00:31.340 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114800.log 2023-10-14T12:00:31.340 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.116954.log: /var/log/ceph/ceph-client.admin.114800.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.116954.log.gz 2023-10-14T12:00:31.340 INFO:teuthology.orchestra.run.smithi078.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.114800.log.gz 2023-10-14T12:00:31.341 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109499.log 2023-10-14T12:00:31.341 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117851.log 2023-10-14T12:00:31.341 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.109499.log: /var/log/ceph/ceph-client.admin.117851.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.109499.log.gz 2023-10-14T12:00:31.341 INFO:teuthology.orchestra.run.smithi078.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.117851.log.gz 2023-10-14T12:00:31.342 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113208.log 2023-10-14T12:00:31.342 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118173.log 2023-10-14T12:00:31.342 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.113208.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-14T12:00:31.343 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.118173.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.113208.log.gz 2023-10-14T12:00:31.343 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph.audit.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.118173.log.gz 2023-10-14T12:00:31.343 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-14T12:00:31.344 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115540.log 2023-10-14T12:00:31.344 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108304.log 2023-10-14T12:00:31.344 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.115540.log: /var/log/ceph/ceph-client.admin.108304.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.115540.log.gz 2023-10-14T12:00:31.344 INFO:teuthology.orchestra.run.smithi078.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.108304.log.gz 2023-10-14T12:00:31.345 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115356.log 2023-10-14T12:00:31.345 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115886.log 2023-10-14T12:00:31.345 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.115356.log: /var/log/ceph/ceph-client.admin.115886.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.115356.log.gz 2023-10-14T12:00:31.345 INFO:teuthology.orchestra.run.smithi078.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.115886.log.gz 2023-10-14T12:00:31.346 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116862.log 2023-10-14T12:00:31.346 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115932.log 2023-10-14T12:00:31.346 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.116862.log: /var/log/ceph/ceph-client.admin.115932.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.116862.log.gz 2023-10-14T12:00:31.347 INFO:teuthology.orchestra.run.smithi078.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.115932.log.gz 2023-10-14T12:00:31.347 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118127.log 2023-10-14T12:00:31.347 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110924.log 2023-10-14T12:00:31.347 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.118127.log: /var/log/ceph/ceph-client.admin.110924.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.118127.log.gz 2023-10-14T12:00:31.348 INFO:teuthology.orchestra.run.smithi078.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.110924.log.gz 2023-10-14T12:00:31.348 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113940.log 2023-10-14T12:00:31.348 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114708.log 2023-10-14T12:00:31.349 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.113940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117435.log 2023-10-14T12:00:31.349 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.114708.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.113940.log.gz 2023-10-14T12:00:31.349 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.117435.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.114708.log.gz 2023-10-14T12:00:31.349 INFO:teuthology.orchestra.run.smithi078.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.117435.log.gz 2023-10-14T12:00:31.350 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115264.log 2023-10-14T12:00:31.350 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108673.log 2023-10-14T12:00:31.350 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.115264.log: /var/log/ceph/ceph-client.admin.108673.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.115264.log.gz 2023-10-14T12:00:31.351 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.108673.log.gz 2023-10-14T12:00:31.351 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117989.log 2023-10-14T12:00:31.351 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109201.log 2023-10-14T12:00:31.351 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.117989.log: /var/log/ceph/ceph-client.admin.109201.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.117989.log.gz 2023-10-14T12:00:31.352 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.109201.log.gz 2023-10-14T12:00:31.352 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112851.log 2023-10-14T12:00:31.352 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116541.log 2023-10-14T12:00:31.353 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.112851.log: /var/log/ceph/ceph-client.admin.116541.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.112851.log.gz 2023-10-14T12:00:31.353 INFO:teuthology.orchestra.run.smithi078.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.116541.log.gz 2023-10-14T12:00:31.353 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113664.log 2023-10-14T12:00:31.354 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117759.log 2023-10-14T12:00:31.354 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.113664.log: /var/log/ceph/ceph-client.admin.117759.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.113664.log.gz 2023-10-14T12:00:31.354 INFO:teuthology.orchestra.run.smithi078.stderr: 95.9% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2023-10-14T12:00:31.354 INFO:teuthology.orchestra.run.smithi078.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.117759.log.gz 2023-10-14T12:00:31.355 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109868.log 2023-10-14T12:00:31.355 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110075.log 2023-10-14T12:00:31.355 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.109868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115494.log 2023-10-14T12:00:31.356 INFO:teuthology.orchestra.run.smithi078.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.109868.log.gz 2023-10-14T12:00:31.356 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.110075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109860.log 2023-10-14T12:00:31.356 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114846.log 2023-10-14T12:00:31.356 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.115494.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.110075.log.gz 2023-10-14T12:00:31.357 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.109860.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.115494.log.gz 2023-10-14T12:00:31.357 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.114846.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.109860.log.gz 2023-10-14T12:00:31.357 INFO:teuthology.orchestra.run.smithi078.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.114846.log.gz 2023-10-14T12:00:31.358 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112068.log 2023-10-14T12:00:31.358 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115724.log 2023-10-14T12:00:31.358 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.112068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112252.log 2023-10-14T12:00:31.358 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.112068.log.gz 2023-10-14T12:00:31.359 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.115724.log: /var/log/ceph/ceph-client.admin.112252.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.112252.log.gz 2023-10-14T12:00:31.359 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 91.1%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.115724.log.gz/var/log/ceph/ceph-client.admin.113346.log 2023-10-14T12:00:31.359 INFO:teuthology.orchestra.run.smithi078.stderr: 2023-10-14T12:00:31.360 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114662.log 2023-10-14T12:00:31.360 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.113346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114892.log 2023-10-14T12:00:31.360 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.114662.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.113346.log.gz 2023-10-14T12:00:31.360 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.114662.log.gz 2023-10-14T12:00:31.361 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.114892.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.114892.log.gz 2023-10-14T12:00:31.361 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118709.log 2023-10-14T12:00:31.361 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116657.log 2023-10-14T12:00:31.362 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.118709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110970.log 2023-10-14T12:00:31.362 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.116657.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.118709.log.gz 2023-10-14T12:00:31.362 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.110970.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.116657.log.gz 2023-10-14T12:00:31.362 INFO:teuthology.orchestra.run.smithi078.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.110970.log.gz 2023-10-14T12:00:31.363 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118081.log 2023-10-14T12:00:31.363 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-14T12:00:31.363 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.118081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110234.log 2023-10-14T12:00:31.364 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-osd.1.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.118081.log.gz 2023-10-14T12:00:31.364 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112758.log 2023-10-14T12:00:31.364 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.110234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109496.log 2023-10-14T12:00:31.364 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.110234.log.gz 2023-10-14T12:00:31.365 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.112758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117666.log 2023-10-14T12:00:31.365 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.109496.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.112758.log.gz 2023-10-14T12:00:31.365 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.117666.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.109496.log.gz 2023-10-14T12:00:31.366 INFO:teuthology.orchestra.run.smithi078.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.117666.log.gz 2023-10-14T12:00:31.366 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113756.log 2023-10-14T12:00:31.366 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112989.log 2023-10-14T12:00:31.366 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.113756.log: /var/log/ceph/ceph-client.admin.112989.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.113756.log.gz 2023-10-14T12:00:31.367 INFO:teuthology.orchestra.run.smithi078.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.112989.log.gz 2023-10-14T12:00:31.367 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114522.log 2023-10-14T12:00:31.367 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110227.log 2023-10-14T12:00:31.368 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.114522.log: /var/log/ceph/ceph-client.admin.110227.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.114522.log.gz 2023-10-14T12:00:31.368 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.110227.log.gz 2023-10-14T12:00:31.368 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115632.log 2023-10-14T12:00:31.369 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116309.log 2023-10-14T12:00:31.369 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.115632.log: /var/log/ceph/ceph-client.admin.116309.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.115632.log.gz 2023-10-14T12:00:31.369 INFO:teuthology.orchestra.run.smithi078.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.116309.log.gz 2023-10-14T12:00:31.369 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.97592.log 2023-10-14T12:00:31.370 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112390.log 2023-10-14T12:00:31.370 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph.tmp-client.admin.97592.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.97592.log.gz 2023-10-14T12:00:31.370 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.112390.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.112390.log.gz 2023-10-14T12:00:31.371 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113526.log 2023-10-14T12:00:31.371 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117297.log 2023-10-14T12:00:31.371 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.113526.log: /var/log/ceph/ceph-client.admin.117297.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.113526.log.gz 2023-10-14T12:00:31.372 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.117297.log.gz 2023-10-14T12:00:31.372 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114616.log 2023-10-14T12:00:31.372 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114476.log 2023-10-14T12:00:31.372 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.114616.log: /var/log/ceph/ceph-client.admin.114476.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.114616.log.gz 2023-10-14T12:00:31.373 INFO:teuthology.orchestra.run.smithi078.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.114476.log.gz 2023-10-14T12:00:31.373 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104165.log 2023-10-14T12:00:31.373 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112344.log 2023-10-14T12:00:31.373 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.104165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109500.log 2023-10-14T12:00:31.374 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.112344.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.104165.log.gz 2023-10-14T12:00:31.374 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.109500.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.112344.log.gz 2023-10-14T12:00:31.374 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115586.log 2023-10-14T12:00:31.375 INFO:teuthology.orchestra.run.smithi078.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.109500.log.gz 2023-10-14T12:00:31.375 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110786.log 2023-10-14T12:00:31.375 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.115586.log: /var/log/ceph/ceph-client.admin.110786.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.115586.log.gz 2023-10-14T12:00:31.376 INFO:teuthology.orchestra.run.smithi078.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.110786.log.gz 2023-10-14T12:00:31.376 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111978.log 2023-10-14T12:00:31.376 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112943.log 2023-10-14T12:00:31.376 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.111978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116908.log 2023-10-14T12:00:31.377 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.112943.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.111978.log.gz 2023-10-14T12:00:31.377 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.116908.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.112943.log.gz 2023-10-14T12:00:31.377 INFO:teuthology.orchestra.run.smithi078.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.116908.log.gz 2023-10-14T12:00:31.377 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117805.log 2023-10-14T12:00:31.378 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116611.log 2023-10-14T12:00:31.378 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.117805.log: /var/log/ceph/ceph-client.admin.116611.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.117805.log.gz 2023-10-14T12:00:31.378 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118219.log 2023-10-14T12:00:31.379 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.116611.log.gz 2023-10-14T12:00:31.379 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110231.log 2023-10-14T12:00:31.379 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.118219.log: /var/log/ceph/ceph-client.admin.110231.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.118219.log.gz 2023-10-14T12:00:31.379 INFO:teuthology.orchestra.run.smithi078.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.110231.log.gz 2023-10-14T12:00:31.380 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108963.log 2023-10-14T12:00:31.380 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113480.log 2023-10-14T12:00:31.380 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.108963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113894.log 2023-10-14T12:00:31.380 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.113480.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.108963.log.gz 2023-10-14T12:00:31.381 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.113894.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.113480.log.gz 2023-10-14T12:00:31.381 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.113894.log.gz 2023-10-14T12:00:31.381 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108915.log 2023-10-14T12:00:31.382 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118495.log 2023-10-14T12:00:31.382 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.108915.log: /var/log/ceph/ceph-client.admin.118495.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.108915.log.gz 2023-10-14T12:00:31.382 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.118495.log.gz 2023-10-14T12:00:31.382 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111154.log 2023-10-14T12:00:31.383 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115678.log 2023-10-14T12:00:31.383 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.111154.log: /var/log/ceph/ceph-client.admin.115678.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.115678.log.gz 2023-10-14T12:00:31.383 INFO:teuthology.orchestra.run.smithi078.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.111154.log.gz 2023-10-14T12:00:31.383 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111062.log 2023-10-14T12:00:31.384 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117527.log 2023-10-14T12:00:31.384 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.111062.log: /var/log/ceph/ceph-client.admin.117527.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.111062.log.gz 2023-10-14T12:00:31.384 INFO:teuthology.orchestra.run.smithi078.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.117527.log.gz 2023-10-14T12:00:31.385 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103738.log 2023-10-14T12:00:31.385 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110510.log 2023-10-14T12:00:31.385 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.103738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108769.log 2023-10-14T12:00:31.385 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.110510.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.103738.log.gz 2023-10-14T12:00:31.386 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.108769.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.110510.log.gz 2023-10-14T12:00:31.386 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.108769.log.gz 2023-10-14T12:00:31.386 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115218.log 2023-10-14T12:00:31.386 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109866.log 2023-10-14T12:00:31.387 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.115218.log: /var/log/ceph/ceph-client.admin.109866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109247.log 2023-10-14T12:00:31.387 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.115218.log.gz 2023-10-14T12:00:31.387 INFO:teuthology.orchestra.run.smithi078.stderr:gzip 88.5%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.109866.log.gz-- 2023-10-14T12:00:31.388 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.112298.log 2023-10-14T12:00:31.388 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.109247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109859.log 2023-10-14T12:00:31.388 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.112298.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.109247.log.gz 2023-10-14T12:00:31.388 INFO:teuthology.orchestra.run.smithi078.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.112298.log.gz 2023-10-14T12:00:31.389 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109101.log 2023-10-14T12:00:31.390 INFO:teuthology.orchestra.run.smithi195.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-10-14T12:00:31.390 INFO:teuthology.orchestra.run.smithi195.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-10-14T12:00:31.395 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.109859.log: /var/log/ceph/ceph-client.admin.109101.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.109859.log.gz 2023-10-14T12:00:31.396 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.109101.log.gz 2023-10-14T12:00:31.396 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-14T12:00:31.396 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114235.log 2023-10-14T12:00:31.396 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.114235.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.114235.log.gz 2023-10-14T12:00:31.397 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112805.log 2023-10-14T12:00:31.397 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112482.log 2023-10-14T12:00:31.397 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.112805.log: 88.3%/var/log/ceph/ceph-client.admin.112482.log: -- replaced with /var/log/ceph/ceph-client.admin.112805.log.gz 2023-10-14T12:00:31.398 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.112482.log.gz 2023-10-14T12:00:31.398 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108627.log 2023-10-14T12:00:31.398 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113572.log 2023-10-14T12:00:31.398 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.108627.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.108627.log.gz 2023-10-14T12:00:31.399 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.113572.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.113572.log.gz 2023-10-14T12:00:31.399 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118665.log 2023-10-14T12:00:31.399 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109498.log 2023-10-14T12:00:31.400 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.118665.log: 87.5%/var/log/ceph/ceph-client.admin.109498.log: -- replaced with /var/log/ceph/ceph-client.admin.118665.log.gz 2023-10-14T12:00:31.400 INFO:teuthology.orchestra.run.smithi078.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.109498.log.gz 2023-10-14T12:00:31.400 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115840.log 2023-10-14T12:00:31.400 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108389.log 2023-10-14T12:00:31.401 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.115840.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.115840.log.gz 2023-10-14T12:00:31.401 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.108389.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.108389.log.gz 2023-10-14T12:00:31.401 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-10-14T12:00:31.401 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109147.log 2023-10-14T12:00:31.402 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-mgr.y.log: /var/log/ceph/ceph-client.admin.109147.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.109147.log.gz 2023-10-14T12:00:31.402 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116495.log 2023-10-14T12:00:31.402 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.116495.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.116495.log.gz 2023-10-14T12:00:31.403 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112022.log 2023-10-14T12:00:31.403 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.112022.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.112022.log.gz 2023-10-14T12:00:31.403 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117343.log 2023-10-14T12:00:31.403 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.117343.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.117343.log.gz 2023-10-14T12:00:31.404 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114568.log 2023-10-14T12:00:31.404 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.114568.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.114568.log.gz 2023-10-14T12:00:31.404 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116010.log 2023-10-14T12:00:31.404 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.116010.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.116010.log.gz 2023-10-14T12:00:31.405 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114281.log 2023-10-14T12:00:31.405 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.114281.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.114281.log.gz 2023-10-14T12:00:31.405 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117389.log 2023-10-14T12:00:31.405 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.117389.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.117389.log.gz 2023-10-14T12:00:31.406 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112114.log 2023-10-14T12:00:31.406 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.112114.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.112114.log.gz 2023-10-14T12:00:31.406 INFO:teuthology.orchestra.run.smithi078.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-10-14T12:00:31.563 INFO:teuthology.orchestra.run.smithi078.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-14T12:00:31.736 INFO:teuthology.orchestra.run.smithi078.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-14T12:00:31.782 INFO:teuthology.orchestra.run.smithi195.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-10-14T12:00:31.788 INFO:teuthology.orchestra.run.smithi195.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-10-14T12:00:31.795 INFO:teuthology.orchestra.run.smithi078.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-14T12:00:31.801 INFO:teuthology.orchestra.run.smithi195.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-10-14T12:00:31.822 INFO:teuthology.orchestra.run.smithi078.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-14T12:00:31.827 INFO:teuthology.orchestra.run.smithi195.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-10-14T12:00:31.829 INFO:teuthology.orchestra.run.smithi195.stderr: 2023-10-14T12:00:31.829 INFO:teuthology.orchestra.run.smithi195.stderr:real 0m0.598s 2023-10-14T12:00:31.830 INFO:teuthology.orchestra.run.smithi195.stderr:user 0m2.315s 2023-10-14T12:00:31.830 INFO:teuthology.orchestra.run.smithi195.stderr:sys 0m0.128s 2023-10-14T12:00:31.873 INFO:teuthology.orchestra.run.smithi078.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-10-14T12:00:31.875 INFO:teuthology.orchestra.run.smithi078.stderr: 2023-10-14T12:00:31.875 INFO:teuthology.orchestra.run.smithi078.stderr:real 0m0.670s 2023-10-14T12:00:31.876 INFO:teuthology.orchestra.run.smithi078.stderr:user 0m2.629s 2023-10-14T12:00:31.876 INFO:teuthology.orchestra.run.smithi078.stderr:sys 0m0.188s 2023-10-14T12:00:31.876 INFO:tasks.ceph:Archiving logs... 2023-10-14T12:00:31.877 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/log/ceph to /home/teuthworker/archive/rishabh-2023-10-13_18:04:14-fs-rishabh-2023oct13-testing-default-smithi/7425292/remote/smithi078/log 2023-10-14T12:00:31.878 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-14T12:00:32.466 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/log/ceph to /home/teuthworker/archive/rishabh-2023-10-13_18:04:14-fs-rishabh-2023oct13-testing-default-smithi/7425292/remote/smithi195/log 2023-10-14T12:00:32.467 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-14T12:00:32.832 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-14T12:00:32.862 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-14T12:00:32.863 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-14T12:00:32.866 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-14T12:00:32.943 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-14T12:00:32.954 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-14T12:00:32.954 DEBUG:teuthology.orchestra.run.smithi078:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-14T12:00:32.957 DEBUG:teuthology.orchestra.run.smithi195:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-14T12:00:32.971 INFO:teuthology.orchestra.run.smithi078.stderr:bash: ntpq: command not found 2023-10-14T12:00:32.973 INFO:teuthology.orchestra.run.smithi195.stderr:bash: ntpq: command not found 2023-10-14T12:00:32.976 INFO:teuthology.orchestra.run.smithi078.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-14T12:00:32.976 INFO:teuthology.orchestra.run.smithi078.stdout:=============================================================================== 2023-10-14T12:00:32.976 INFO:teuthology.orchestra.run.smithi078.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 63 +2473us[+2476us] +/- 99ms 2023-10-14T12:00:32.977 INFO:teuthology.orchestra.run.smithi078.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 61 -1235us[-1235us] +/- 80ms 2023-10-14T12:00:32.977 INFO:teuthology.orchestra.run.smithi078.stdout:^* hv03.front.sepia.ceph.com 3 6 377 63 -604us[ -600us] +/- 66ms 2023-10-14T12:00:32.977 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-10-14T12:00:32.979 INFO:teuthology.orchestra.run.smithi195.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-14T12:00:32.979 INFO:teuthology.orchestra.run.smithi195.stdout:=============================================================================== 2023-10-14T12:00:32.980 INFO:teuthology.orchestra.run.smithi195.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 62 +2494us[+2494us] +/- 99ms 2023-10-14T12:00:32.980 INFO:teuthology.orchestra.run.smithi195.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 62 -1229us[-1229us] +/- 80ms 2023-10-14T12:00:32.980 INFO:teuthology.orchestra.run.smithi195.stdout:^* hv03.front.sepia.ceph.com 3 6 377 63 -614us[ -630us] +/- 66ms 2023-10-14T12:00:32.980 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-10-14T12:00:32.981 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-14T12:00:32.992 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-14T12:00:32.993 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-14T12:00:33.007 DEBUG:teuthology.orchestra.run.smithi078:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-14T12:00:33.115 DEBUG:teuthology.orchestra.run.smithi195:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-14T12:00:33.207 DEBUG:teuthology.orchestra.run.smithi078:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-14T12:00:33.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T12:00:33.240 DEBUG:teuthology.orchestra.run.smithi195:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-14T12:00:33.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T12:00:33.272 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-14T12:00:33.283 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-14T12:00:33.294 INFO:teuthology.task.internal:Duration was 855.934377 seconds 2023-10-14T12:00:33.295 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-14T12:00:33.308 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-14T12:00:33.308 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-14T12:00:33.311 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-14T12:00:33.355 INFO:teuthology.orchestra.run.smithi078.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-14T12:00:33.359 INFO:teuthology.orchestra.run.smithi195.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-14T12:00:33.506 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x has been restored 2023-10-14T12:00:33.601 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-14T12:00:33.601 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi078.front.sepia.ceph.com 2023-10-14T12:00:33.602 DEBUG:teuthology.orchestra.run.smithi078:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-14T12:00:33.623 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi195.front.sepia.ceph.com 2023-10-14T12:00:33.623 DEBUG:teuthology.orchestra.run.smithi195:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-14T12:00:33.644 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-14T12:00:33.644 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-14T12:00:33.664 DEBUG:teuthology.orchestra.run.smithi195:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-14T12:00:33.724 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-14T12:00:33.725 DEBUG:teuthology.orchestra.run.smithi078:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-14T12:00:33.746 DEBUG:teuthology.orchestra.run.smithi195:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-14T12:00:33.852 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-14T12:00:33.862 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-14T12:00:33.863 DEBUG:teuthology.orchestra.run.smithi078:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-14T12:00:33.890 DEBUG:teuthology.orchestra.run.smithi195:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-14T12:00:33.917 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-14T12:00:33.929 DEBUG:teuthology.orchestra.run.smithi078:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-14T12:00:33.933 DEBUG:teuthology.orchestra.run.smithi195:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-14T12:00:33.956 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = core 2023-10-14T12:00:33.983 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern = core 2023-10-14T12:00:33.996 DEBUG:teuthology.orchestra.run.smithi078:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-14T12:00:34.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T12:00:34.026 DEBUG:teuthology.orchestra.run.smithi195:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-14T12:00:34.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T12:00:34.052 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-14T12:00:34.063 INFO:teuthology.task.internal:Transferring archived files... 2023-10-14T12:00:34.064 DEBUG:teuthology.misc:Transferring archived files from smithi078:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2023-10-13_18:04:14-fs-rishabh-2023oct13-testing-default-smithi/7425292/remote/smithi078 2023-10-14T12:00:34.064 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-14T12:00:34.103 DEBUG:teuthology.misc:Transferring archived files from smithi195:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2023-10-13_18:04:14-fs-rishabh-2023oct13-testing-default-smithi/7425292/remote/smithi195 2023-10-14T12:00:34.104 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-14T12:00:34.140 INFO:teuthology.task.internal:Removing archive directory... 2023-10-14T12:00:34.141 DEBUG:teuthology.orchestra.run.smithi078:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-14T12:00:34.143 DEBUG:teuthology.orchestra.run.smithi195:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-14T12:00:34.191 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-14T12:00:34.203 INFO:teuthology.task.internal:Not uploading archives. 2023-10-14T12:00:34.204 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-14T12:00:34.215 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-14T12:00:34.216 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-14T12:00:34.218 DEBUG:teuthology.orchestra.run.smithi195:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-14T12:00:34.232 INFO:teuthology.orchestra.run.smithi078.stdout: 262307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 14 12:00 /home/ubuntu/cephtest 2023-10-14T12:00:34.248 INFO:teuthology.orchestra.run.smithi195.stdout: 262307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 14 12:00 /home/ubuntu/cephtest 2023-10-14T12:00:34.249 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-14T12:00:34.262 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-14T12:00:34.286 INFO:teuthology.nuke:Checking targets against current locks 2023-10-14T12:00:34.311 DEBUG:teuthology.nuke:shortname: smithi078 2023-10-14T12:00:34.312 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-14T12:00:34.332 DEBUG:teuthology.nuke:shortname: smithi195 2023-10-14T12:00:34.332 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-14T12:00:34.336 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi078.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-10-13_18:04:14-fs-rishabh-2023oct13-testing-default-smithi/7425292', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-14 11:40:59.830502', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-10-14T12:00:34.360 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi195.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-10-13_18:04:14-fs-rishabh-2023oct13-testing-default-smithi/7425292', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-14 11:40:59.832506', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-10-14T12:00:34.390 INFO:teuthology.orchestra.console.smithi078:Power off 2023-10-14T12:00:34.391 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-14T12:00:34.414 DEBUG:teuthology.orchestra.console.smithi078:power off output: Chassis Power Control: Down/Off 2023-10-14T12:00:34.415 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-14T12:00:34.436 INFO:teuthology.orchestra.console.smithi195:Power off 2023-10-14T12:00:34.437 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-14T12:00:34.447 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on 2023-10-14T12:00:34.460 DEBUG:teuthology.orchestra.console.smithi195:power off output: Chassis Power Control: Down/Off 2023-10-14T12:00:34.461 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-14T12:00:34.483 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is on 2023-10-14T12:00:38.448 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-14T12:00:38.484 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-14T12:00:38.559 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on 2023-10-14T12:00:38.596 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is on 2023-10-14T12:00:42.562 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-14T12:00:42.598 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-14T12:00:42.675 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is off 2023-10-14T12:00:42.710 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is off 2023-10-14T12:00:42.777 INFO:teuthology.orchestra.console.smithi078:Power off completed 2023-10-14T12:00:42.811 INFO:teuthology.orchestra.console.smithi195:Power off completed 2023-10-14T12:00:42.916 INFO:teuthology.run:Summary data: description: fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} duration: 855.9343774318695 failure_reason: 'Test failure: test_join_fs_drop (tasks.cephfs.test_failover.TestClusterAffinity)' flavor: default owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=d85300d859db4d299817149550aafc40 status: fail success: false 2023-10-14T12:00:42.916 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-14T12:00:42.995 INFO:teuthology.run:FAIL